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Prefatory Note 

In Work Unit TEXTRUCT, the Human Resources Rest arch 
Organization undertook a program of research directed toward the 
development of efficient and effective instructional techniques that 
would be applicable to the extensive program of Army technical 
training. The research took place during the years 1958-64; it was 
conducted at HumRRO Division No. 5, at Fort Bliss, Texas. 

The papers in this collection were presented at military and 
professional meetings or appeared in professional journals during 
the course of the research or Work Unit TEXTRUCT. Other publi- 
cation s under this Work Unit are listed at the back of this publication . 

Because of the continuing relevance of the subject matter of 
these papers, they are being issued in a group as part 0 f the 
HumRRO Professional Paper series. This series was initiated in 
order to provide permanent record of specialized aspects of HumRRO 
work, and deposit in the scientific a'.d technical information storage 
and retrieval systems cf the Department of Defense and the National 
Technical Information Service. 
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TEACHING: TODAY AND TOMORROW 1 

Robert F . Mager 



In this paper some principles of learning and some 
facts about the teacher-learner situation in the 
modern setting are discussed. Purposes and methods 
of automating instruction so that students will be 
able to learn more — and teachers teach better — are 
presented. There are descriptions of several 
teaching devices, 



During the past 50 years we have seen tremendous advances in man> 
disciplines. We have seen communication progress from the crystal se. 
and the telegraph key to color television and microwave. Data process 
ing has moved from the simple adding machine to huge electronic com- 
puters that can handle millions of bits of information and perform 
millions of operations per second. 

The tools of astronomy have blossomed from the simple optical 
telescope to the huge radio telescopes that look far out into the 
universe; and aviation has exploded from nothingness into supersonic 
flight, space ships and satellites. 

With each new development, with each advance, has come an increase 
in the number of facts and in the number of equations to be learned. 

As the mountains of knowledge grow higher it becomes increasingly 
necessary for the student to assimilate greater amounts of material 
before he can be considered competent in his chosen field. 

The young science of learning has alsc made progress during the 
past 50 years, although seemingly insignificant when compared with the 
great strides made in the areas mentioned above. But even the progress 
which has been made in the science of learning has not been applied to 
the learning environment. The technology of learning has not kept pace 
with the science. 

The present techniques of education have long been inadequate and 
become even more noticeably cumbersome as the amount of information 
which must be mastered by the student increases. We have approached 
the point where we can no longer afford the luxury of an outdated 
technology of learning. 

Though w f e do not pretend to know a good deal about the learning 
process, chere is, nevertheless, a vast chasm between what we know 



*This article appeared in IRE Student Quarterly, September 1959. 
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about learning and what we do about it. There are several principles 
of learning which have been sufficiently validated by research which, 
if applied, would increase learning efficiency. There is a fairly 
respectable body of knowledge which, if utilized, would undoubtedly 
make a startling difference in the amount of material which could be 
mastered per unit time. 

SOME PRINCIPLES OF LEARNING 

Here, in as nontechnical terms as possible, is a description of 
some of the learning principles which are known but which, for the 
most part, have been left to lie idle. 

PRINCIPLE OF PARTICIPATION: This principle explains that learning is 

more efficient when the learner is called upon to make frequent 
responses relevant to the skill being learned; that is,, when the 
learner participates in the learning. More simply, we learn by doing. 

Example: If you were watching a film on the tuning of an IF strip 
you would learn more if you were required to answer questions during 
the film than you would if you simply relaxed while watching the film 
from beginning to end. You would learn even better by actually tuning 
an IF strip. 

PRINCIPLE OF KNOWLEDGE OF RESULTS: Learning is more efficient when the 

learner is apprised of the accuracy or appropriateness of his responses, 
when he receives feedback as to how well he is doing. 

Example: If you were trying to learn to fire a rifle your accuracy 

would probably not improve unless you could see where your shots 
were hitting. 

PRINCIPLE OF IMMEDIACY OF KNOWLEDGE OF RESULT: For knowledge of results 

to be mo:t effective it must follow immediately after a response. In 
other v:oids, we learn by doing, and by knowing right now how we did, 

The term ‘'immediate" here means within a few seconds after the response. 

Example: You take an examination, and two or three days later your 

paper is returned. Though such a procedure is quite legitimate when 
the exam is used for the purpose of determining your progress, it is all 
but useless when the exam is used as a learning device. 

PRINCIPLE. OF REINFORCEMENT: Learning is more effective when the learner 

is rewarded (more technically, reinforced) for correct responses. This 
is a very tricky principle to apply, however, because it is frequently 
difficult to know just what is reinforcing to a particular individual; 
what is rewarding to one may not be rewarding to another. 

Example: After being a poor student for most of the semester in 

mechanical drawing you finally produce a good-looking piece of work. 

The instructor does handsprings and shows your work proudly to his 
colleague. The probability increases that your mechanical drawing 
skill will improve, 

INDIVIDUAL DIFFERENCES: It h$': been a truism in psychology for many 

years that there are individual differences. People differ in their 
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bf.sic intelligence, in their skills and abilities, and they differ in 
the rate at which they learn. The structure of most of our educa- 
tional system ignores this basic fact of human behavior, however, in 
that a single instructor is expected to fill simultaneously the needs 
of large numbers of different and differing individuals. But as we 
shall presently see, none of the principles discussed are applied 
very well in this traditional environment. Regardless of the instruc- 
tor's skill or enthusiasm the very structure of tae learning environ- 
ment stacks the deck against him— and the student. 

With a single teacher and a single learner these principles can be 
applied with some degree of efficiency. To see how this can occur it 
will be useful to consider what happens in the teacher- learner situation. 



THE TEACHER-LEARNER SITUATION 



The teacher- learner situation is very much like a transmitter- 
receiver system, where the object is to transfer information from the 
transmitting unit to the receiving unit with as little loss and dis- 
tortion as possible. Kith electronic systems this goal can be achieved 
rather readily. In servo and selsyn systems a high degree of informa- 
tion transfer can bs accomplished through relatively simple circuitry, 
and such transmission can be made even more efficient when a feedback 
loop is employed. With human systems we are not quite so lucky, Much 
information is lost during transmission because of such factors as 
inattention, motivation— or the lack of it — and inappropriate trans- 
mission rate. Because of this state of affairs provision for feedback 
is even more essential, feedback which will allow the teacher to deter- 
mine whether information was received and how accurately it was received, 
every step of the way. 

With a single student the teacher can provide information to the 
student, an item at a time, and then check to see if the student r, got it** 
and whether he got it correctly. On the basis of this feedback the 
instructor can then determine what he should tell the student next. He 
can modify his teaching program to fit the needs of the learner. 

As the learner makes each response, the teacher can provide him with 
immediate feedback in the form of knowledge of results. He can reinforce 
correct responses (by telling the student he is right), and he can 
decrease the probability of an error being repeated by withholding rein- 
forcement when errors occur and by providing corrective information. 

Thus, in this kind of situation, the student participates actively in 
the learning, receives immediate knowledge of results after each response, 
is reinforced for correct responses, and is allowed to proceed at the 
rate best for him. 



But a description of the single teacher- learner situation is not a 
description of the most commonly used learning environment. We are very 
seldom in a learning situation where we enjoy such attention. In fact, 
harking back to our transmitter-receiver analogy, we find that we usually 
have a situation wherein a single transmitter is beaming information at 
anywhere from 10 to 250 receivers, each of which has band-pass filters 
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tuned to drop out different bits of information (individual differences 
again). A best, a single transmission rate will be optimal for only a 
small percentage of the receivers. 

In the classroom as we know it, it is very difficult for a single 
instructor to arrange for students to make frequent relevant responses, 
and it would be next to impossible for this single instructor to pro- 
vide selective knowledge of results even if students were participating 
frequently. So even if efficient feedback were possible to achieve in 
the classroom the best the instructor could do is maximize his instruc- 
tion for only a small proportion of students. 

It is almost impossible to effectively apply most of the principles 
of learning in the classroom as we know it. Not because the instructor 
is inadequate or unenthusiastic (though the signal -to-noise ratio of 
some transmissions has been known to approach one-to-one, but s imply 
because the configuration of the learning environment does not allow 
these principles to be applied. Another way of saying this is that 
even though we are aware of such fundamental behavioral facts as indi- 
vidual differences, we don’t do very much about it. Even though we 
have known for a long time that people learn at different rates, and 
even though we know that to be most effective the learning environment 
must be tailored to the individual, we still put large groups of stu- 
dents in front of a single instructor and insist that they all learn 
at the same rate. Though this may be a convenient way of doing things, 
or economically necessary, it isn’t very efficient. 

THE PROMISE OF AUTOMATION 

Such a situation need not be perpetuated. New methods of instruc- 
tion are under development which promise co provide students with the 
means whereby they can learn more things in less time, and which promis 
to provide the instructor with long overdue relief from the more menial 
and monotonous aspects of his job. 

The methods to which 1 refer are those involving the automation of 
instruction and the use of teaching machines (more accurately, teaching 
devices). Such devices are not mere training aids or audiovisual 
gadgets, nor are they mechanical oddities which will produce regiments 
of hunan robots. Rather, they are devices which, like private tutors, 
carefully present information to the individual learner in small steps, 
require him to participate in some way in the learning, provide the 
learner with immediate knowledge of results, and allow each learner to 
proceed at the rate best for him. In other words, a teaching machine 
is a device which has built into it a relatively efficient feedback 
system; the device and the learner constitute a closed system,. The 
device provides feedback to the student which tells him how well he 
is doing every step of the way, and the learner’s responses provide 
feedback to the device on the basis of which it can modify its program 
so as to provide the learner with exactly that information or guidance 
he needs next. 
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The effort to automate instruction is not new. You may be as sur- 
prised as I was to learn that the first teaching machine was patented 
by Halcyon Skinnar as far back as 1866, and his device appears to be 
about as good as some in use today. (A copy of this patent. No. 72758, 
may be obtained from the U.S. Patent -Office for 25 cents.) Apparently 
little came of his efforts and nothing much happened until 1924, when 
Dr. Pressey of Ohio State University develop ud some devices which 
would test and teach. Again the world wasn’ ; ready and after several 
years without any encouragement whatsoever Dr. Pressey turned to 
other pursuits. Today, however, an ever increasing number of 
researchers and educators are becoming actively interested in the 
automation of instruction. 

Researchers who are working to develop the principles of instruc- 
tional programming and the devices through vhich these programs are 
presented are creating a learning environment which will be more 
efficient than that of the traditional classroom, because it will be 
an environment incorporating the science of learning. These researchers 
are developing a technology of learning based on the science of learning, 

Though there are relatively little data describing results of teach- 
ing by automation, those which are available are highly encouraging. 
Briefly, a first semester German course has been successfully taught 
by a teaching device in approximately 40 hours, and a college algebra 
course has been taught in 30 hours. When tie students of an average 
college psychology class took the course by teaching device the stand- 
aid final exams had to be revised; the median final score was 94% and 
no student made less than 85% on the exam. 

We have been trying to explore the limits of teaching devices and 
learn about programming techniques by giving college algebra problems 
to 11- and 12-year-old children. Even when this material is presented 
by a device as simple as a specially prepared manila folder, we find 
that these students can learn a significant portion of the content. 

Even though these students are generally considered to be below the 
age of ’’readiness" to learn such material as college algebra they are, 
in fact, able to master a significant portion of it when it is appro- 
priately presented in small steps. One of the most unexpected observa- 
tions from this preliminary work came from the youngest subjects, who 
have a rather severe but traditional dislike for the mathematics they 
are being taught at school. Even though the/ were learning such con- 
tent as the associative and distributive law;, absolute numbers, and 
the manipulation of signed numbers, they refused to believe that this 
had anything to do with mathematics because ;:hey could understand it. 

The same sort of thing happened when the concept of the kinetic 
theory of gases was programmed by Dr. Day anf given to juniors in a 
course in physical chemistry at Ohio University. Some of the students 
found the subject matter so easy when presenled in this manner they 
considered it an insult to their proud intelligence. They learned the 
material, you understand, but complained that it was too easy. Appar- 
ently, somebod; has been teaching students that learning can be neither 
pleasant nor simple; whoever it is ought to b2 ashamed of himself, 
because it does not need to be so. 



5 



9 



What the limits are in the effectiveness of programmed instruction 
we don’t yet know. So far, subject matters which have been success- 
fully automated or semi-automated include physics, chemistry, psychology, 
languages, statistics, and art. Though there may be some subjects which 
either cannoc or should not be automated, we do not yet know what 
they are. 



METHODS OF PROGRAMMING 

The automation of instruction requires, first, the preparation of 
the program and, second, construction of the device or medium through 
which the program will be presented. But as important as the device 
may be, the critical aspect is the programming. Without proper prepara- 
tion of the program, without careful breakdown and sequencing of the 
content to be taught, the most beautiful and complicated^ device is of 
little value. 

Let's consider two of the methods of programming and how they might 
be presented to the learner. One method, the sequential program, pre- 
sents information to the student one frame (item) at a time and might, 
for example, require the student to write his response to the frame 
either into the device or on a separate sheet of paper. As soon as he 
does this, the device shows him the correct response (answer), pro- 
viding him with immediate knowledge of results. The second frame (item) 
then appears; again the learner writes his response and again the device 
immediately provides knowledge of results. Frames are constructed and 
sequenced in such a way as to lead the student from a state of ignorance 
to one of complete mastery. With the sequential program the student is 
presented with every item of the sequence, and information is presented 
in such small steps and in such a way as to encourage the learner to 
make correct resporses to almost every frame. 

Here are a few sample frames of a program to teach some facts about 
fractions. Note how the information is presented in very small steps, 
how the learner must respond to each aspect of the material, and how 
the program gradually leads the learner toward an understanding of 
the material. 



1 . 


Fractions can be both common fractions and 
decimal fractions. 2/5 is a common 


fraction 


2. 


1/3 is a fraction. 


common 


3. 


11/16 is a 


common 

fraction 


4. 


The niimber above the dividing line in a common 
fraction is the numerator. In the common 
fraction 2/3, the numerator is 


2 


5. 


In the common fraction 1/3, 1 is the 


numerator 


6. 


In a common fraction the numerator is 
the number 


above the 



dividing line 



7 . 



The number below the dividing lxnc in a 
common fraction is the denominator . In 
the common fraction 2/3 the denominator 
is . 3 



8. 


In the common fraction 11/12, 12 is 
the 




denominator 


9. 


What is the dc lominator of a common 
fraction? 




number below 








dividing line 


10. 


In 6/11 the numerator is fa") 


j 


(a) 6 


and the denominator is (b) 




(b) 11 


11. 


In 17/41 the denominator is fa] 


• 


(a) 41 


and the numerator is fbl 




(b) 17 



How is a sequential program presented to the learner? A simple 
device constructed by the writer for this purpose is shown in Figures 1, 
2, and 3. The frames appear in the window at the left and the learner 
writes his response on a separate sheet of paper. He then lifts the 
flr.p of the "answer" window to check the a (curacy of his response. 

If he is correct, he checks his answer; if he is not, he crosses it 
ou ; and writes it correctly. He then rolls the next frame into view 
wi;h the wheel under his left thumb. As you can see, the "innards" of 
this device look suspiciously like the information rolls found in tube 
testers. More automatic devices have been constructed through which 
this kind of program can be presented, devices which can tabulate the 
nunber of correct and incorrect responses, and which can phase the 
learner forward to more difficult material if he achieves, say, five 
correct items in a row. Less automated devices have also been used, 
since complete automation is not necessary to achieve good results; 
it's the progran that counts. 

Another method, that of alternative programming, samples the stu- 
dents output every step of the way by recuiring him to select an answer 
rather than create one. With an alternative program, the learner is not 
g:.ven every frame of the program. The learner’s response to the device 
determines the information to be presented next. Here, after being 
presented with the first item of information or task, the learner’s 
understanding is tested by a multiple-cho.ce question. He may indicate 
his alternative to the machine by pressing an appropriate button or by 
operating a switch. The device selects the material to be presented 
n?xt on the basis of the learner’s response. If the learner were cor- 
rect the machine might respond, "Your answer was ’impedance.' YOU ARE 
CORRECT, since you saw that both capacity /e and inductive reactances 
are present in this circuit," and then present the next piece of content. 
If an incorrect alternative were chosen by the learner the machine would 
respond in a way calculated to reduce the likelihood of the error being 
repeated, and in a way which would help the learner to understand the 
caterial. If, for example, the learner nade a simple error in calcula- 
tion the device might tell him, "Your an c wer was '50 ohms. 1 Though 
you seem to understand the principle involved here you apparently 



Fig. I. Tfi'i imi r | div'ct m ui«d for prtun^fien of • i«« 
qu«nt>»l program. 



Fig 2. A I* »rri»f £t.wp4f*j Mt riipoma w»fh th* «orra«f on# 
appearing under ffc* fab. 




Fig. 3. Tba m«cFani»m of thii rfavTca ii aiframaly ifmpla. 
Cvrufrucfioft rtirmblfi ffiaf vnd fo fht information 

roffi found in many fuba ftittri. 



added 2 and 2 and got 5, l suggest you slow down a little . o >ou can 
take more care with your calculations. Now let’s try it again. ” If 
the learner’s choice could only have been made on the basis of a wild 
guess he might be told, "One megohm? Now, ccme on. You’re just guess- 
ing. Perhaps we’d better back up a few steps and work our way forward 
more slowly," Here again the learner is carried along at the rate best 
for him. He is active in the learning and is provided with immediate 
knowledge of results. By now you can see that machine teaching is not 
a "mechanical" affair; it is just as personal and intimate as the pro- 
grammer cares to make it. 

What kind of device is used to present the alternative program o 
the learner? Cne which has been constructed used rear-screen micro- 
film projection. The learner indicates his responses to the machine by 
pressing an appropriate button, and the device then selects the frame 
to be presented next and projects it on the screen provided for this 
purpose. Though the devices used to present the alternative program 
are generally more complex than those used to present the sequential 



program, it happens that the alternative program can be presented by a 
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device as simple as a properly constructed book. Suppose, for example, 
that the first iten of information were presented on page 20 and that 
the test item appeared at the bottom of that page. Suppose further 
that each answer to this item were numbered with page numbers. Now, 
when the learner chooses his answer he turns to the page indicated by 
that answer and reads the information appropriate to that response. 

If he is correct he is reinforced and the next item of information is 
presented. If he is incorrect he gets correctional information, fol- 
lowed by another problem. He is given as many explanations and prob- 
lems as are necessary to understand the material, and he is not phased 
forward until he does. This device is sometimes called a "scrambled 
book" or "programmed book" and can be quite an effective teach- 
ing instrument. 



THE NEED FOR DEVICES 

Even though the program is the most important aspect of automated 
instruction, the need for specialize 1 hardware should grow fantastically 
during the next five years, and many engineers will find opportunities 
to make major contributions in another new field. Currently, there is 
a need for research instruments of special design. We have approached 
the point where the limitation in versatility of current devices pre- 
vents explorations of the variety of configurations and situations 
necessary to the development of the most efficient learning environment. 
We need research equipment with sufficient versatility to allow us to 
vary systematically all the variables which affect learning. Such 
devices must provide for the acceptance and recording of different kinds 
of learner responses. A device for teaching electronics would ideally 
allow for presentation of subject matter through several modalities; 
visual, in the form of printed matter, film strips and film clips; 
auditory, in the form of verbal instructions and equipment sounds; 
tactual, so that learners could be made to "feel" the meaning of such 
concepts as "too hot" and "excessive vibration," and possibly even 
olfactory, so that the learner could be presented with the odor of t 
burning transformer. This kind of device will be used for research 
aimed at learning how to maximize the learning situation. 

An entire family of self- testing and review machines is needed. 

Such devices must be capable of being rapidly loaded with as many as 
1,000 test items, must be relatively snail and easy to operate, and 
must be very rugged. Experience with prototypes of these machines 
indicates that when such a self-testing machine is made available to 
students it is used until it is worn out. Why shouldn't there be 
several hundred such machines spotted around a university campus, each 
loaded with test and review items covering different subjects? We know 
that, when available, students will spend a good deal of time "playing" 
such machines; here is a wonderful opportunity to turn the pin-ball 
craze into something profitable for the player too. 

There is need for another family of devices which, rather than 
teach a particular content or skill, vill bv built for the purpose of 
invreasing the proficiency with which ari already learned task will 
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be performed. Such devices will encourage the student to perform faster 
and faster until he has reached some predetermined level of proficiency. 
Skills which might be sharpened and improved by such devices include 
the interpretation of color codes, the accurate reading of meters, the 
use of a slide rule, and the reading of schematics. 

There is a need for hundreds of thousands of small pocket-size or 
highly portable teaching machines for use in connection with ctrrespond- 
ence courses. Most people who sign up for such courses find it very 
difficult to maintain the motivation needed to persevere from one lesson 
to the next, and the use of programmed instruction will not only increase 
the comprehensibility of information presented by correspondence but 
should significantly increase the number of people who complete such 
courses, once begun. I have been preparing material covering the subject 
of instructional programming for non-psychologists (which might be pre- 
sented by correspondence), and 1 find that even though this is an 
embryonic area, much of the material is amenable to one form of pro- 
gramming or another. 

There is another reason why the need for specialized teaching 
devices will expand phenomenally. Up until now researchers in the 
area of programmed instruction have been almost entirely concerned 
with the teaching of what is called ’Verbal skills,” such as the teach- 
ing of mathematics, languages, spelling, psychology, and statistics. 

The entire domain of teaching perceptual and motor skills is virtually 
virgin territory. Special devices will be required for the teaching 
of such perceptual skills is meter reading, scope reading, component 
and color code identification, and for the teaching of such motor 
skills as are involved in test equipment operation and in the opera- 
tion and repair of complex electronic equipments. There is no reason 
why the electronic engineering laboratory cannot now have a separate 
teaching machine to teach the energizing, calibration, operation, and 
uti lization of each standard piece of test equipment. Such devices 
can be built for a cost not exceeding that of a medium-priced oscil- 
loscope. In the future prime contracts calling for the construction 
of complex missile systems, aircraft, and space vehicles will undoubt- 
edly include requirements, not only for training aids and simulators, 
but for devices which will actually teach the operation and mainte- 
nance associated with these equipments. Intriguing possibilities will 
no doubt soon be available to engineers interested in the programing 
of instruction. 

Once programmed instruction has been introduced into the curriculum 
students will be able to learn considerably more material in consid- 
erably less time. But when tnis happens, what will happen to the 
instructor? Will automation put the teacher out of a job? Certainly 
not. But it will change the nature of his work. The advent of. automa- 
tion will give him a breath of fresh air and a new dignity, because 
now, instead of standing in front of an apathetic group of students 
trying vainly to transmit information in an environment ill-suited to 
his purpose, he will take on the role of a consultant; he wili become 
an expert who can give individual attention to those reeding it. He 
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PRELIMINARY STUDIES IN AUTOMATED TEACHING 1 

Robert F. Mager 



Data collected during preliminary studies of two methods of 
automated teaching were presented, accompanied by a description of 
characteristics and programing procedures for each method. 

Automated teaching methods provide several advantages unattainable 
in the traditional classroom; every student advances at his own pace, 
participates actively in the learning, is immediately informed (by the 
device) of the correctness of his responses, and is rewarded for 
correct responses. Techniques of instruction automation ‘provide 
means for completing the information loop between instructor and 
learner by providing feedback from the learner to the "ins true tor," 
thus enabling the teaching device to modify its program so as to 
optimize each student’s rate of progress. 

One method described sequentially presents items of information 
and test to the learner so that facts and concepts are built up and 
solidified to any required degree of learning. Items are scaled in 
such small increments of difficulty thit the learner predominately 
makes correct responses to the machine and is thus maximally rewarded. 

Another method described presents Information in such a way that 
the responsexof the learner directly determines the information to be 
presented next. This programing allow.* fast learners to move rapidly 
ahead unimpeded by unnecessary explanations, while slower learners 
are provided with as many different explanations and practice items 
as are called for by his responses to t:he device. 

Results of preliminary studies of these methods of automating 
instruction are presented. Two "difficult" concepts encountered in 
freshman chemistry were automated and presented to a freshman class 
of chemistry students at a midwestern university. Data indicate the 
superiority of this instruction over stendard classroom methods* 

In another study concepts of semiccnductor physics were programed 
into a teaching device. Two groups of Institute of Radio Engineer 
members unfamiliar with this content were given instruction, one 
group by traditional lecture and the other by teaching device. 

Results of this experiment were reported. 

Finally, results of automated teaching of college algebra to 
; 11- and 12-year-old students, were disejssed. 

i 

f 



1 This is a summary of a paper given at the national convention of 
IRE (Institute of Radio Engineers) in Niw York City, 1959. 
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DEVELOPING NEW INSTRUCTIONAL TECHNIQUES 1 

P.G. Whitmore 



This paper describes research techniques dealing with 
the problems of specifying criteria of training , devel- 
oping procedures for guiding and assessing learning 
during training, and engineering known principles of 
learning into the training context. The emphasis is 
on what should be learned and how it should be learned. 



The phrase "instructional technique" defies precise definition. 
Most of us, for instance, have at one time or another been faced with 
the chore of learning to recite the 12 cranial nerves in order, and 
most of us have probably done so by first 3 earning some mnemonic 
device, which in turn was acquired by rote memorization. The ability 
to recite the names of the 12 cranial nerves in order obviously con- 
stitutes the objective or criterion of this small training program. 
Does the term "instructional technique' refer to the employment of a 
mnemonic device, to the rote memorization of that device, to the 
establishing of associations between the ordered words in the mne- 
monic device and the names of the cranial nerves, or to all three? 

The referents for many of the words currently used to describe the 
training process are either not specified or poorly specified. Any 
research in this area consequently must begin with what might be 
termed an operational analysis of the training process itself. 

This process might best be described as a series of behavioral 
acquisitions beginning with an already acquired repertoire of behav- 
iors and terminating with the acquisition of a specified set of 
behaviors. The first problem that must be resolved during the con- 
struction of a training program is selecting and specifying the 
behaviors whose acquisition the student must be capable of demon- 
strating at the termination of the training process. This problem 
is akin to the traditional content derivation problem, but differs 
from it in one very important respect. In nosl content derivation 
studies, content has been derived and specified in terms of the 
stimuli to which the student is to be exposed during training. Our 
interest, however, is in specifying the behaviors the student must 
in fact acquire. The problem of deriving these behaviors from the 
job requirements in itself comprises a major research effort. It is 
a problem, however, which need not be resolved prior to the initiation 



*Paper for Symposium at annual meeting of Southwestern Psychologi- 
cal Association, Spring 1960. 
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of research on the training process itself. This is not true, however, 
of the problem of developing procedures for specifying behaviors. The 
requirements for specifying the objectives of training must take into 
account the purposes for which the objectives are to be used. 

One action that must invariably occur in the conduct of a training 
program is deciding which students are, and which are not, to be grad- 
uated from the program at its termination. Cureton (1) has pointed out 
that "the de facto aims of an educational program and of every part 
thereof consist of those acts on the basis of which the students and 
the program are in fact evaluated. If any stated aim is not analyzed 
into specific actions and those actions observed and scored and reported 
the statement is no more than empty verbiage. 11 

Amplifying this point Ebel (2) has suggested operationally defining 
educational objectives in terms of an extended series of test problems 
before curricula are designed . In this connection it is interesting to 
note that Adkins (3) has broadly defined a test as 1r a means of drawing 
inferences about persons, based upon their responses to a sampling of 
a field of behavior. " What we are seeking in this phase of the research 
is a means for specifying the procedure by which the terminal pass-fail 
decision in training is to be made. Ebel*s suggestion of doing this by 
means of an extended series of test problems is not a wholly satisfactory 
solution, primarily because the development of an extended series of 
test problems can become an overly tedious process. For some training 
programs there may well be an almost infinite number of test prob- 
lems possible. 

On the other hand, a danger exists in not specifying the objectives 
in sufficiently precise terms. For instance, if one objective of a 
geography class is a statement that a student must "familiarize" him- 
self with a map of the world, nothing has been said about the typ^ of 
behavior that must be observed in order to conclude that this objective 
has or has not been met. Would such an objective require that the 
student be able to produce and label an outline map of the world from 
memory or would it simply require that he Jabol certain geographical 
features on a map presented to him? In either case, what features must 
he be able to label— major land masses, oceans, rivers, lakes, countries, 
capitol cities, major cities, and so forth? There would be almost as 
many different interpretations of such an objective as there were 
instructors teaching the course, Thtro can surely be no purpose in 
expending either time or energy in the derivation of objectives and 
content for training which are subject to such a varied number of 
interpretations. Such a procedure could lead only to capricious eval- 
uation of student achievement. 

Training objectives might better be stated in terms of a description 
of the population of problems from which any particular test is to be 
drawn. Cureton (1) lists five characteristics that should be specified 
in the definition of a to-be-measured function, that is, of a population 
of test problems, as follows: 

(1) "The acts or operations of which it is composed; 

(2) "The materials acted upon; 
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(3) "The situations in which these acts or operations properly 
take place; 

(4) "The results of products of the act or operations, and 

(5) "The particular aspects or features of the acts or of 
their results or products which are to be considered as 
germane to the function." 

In addition, it will be necessary to specify the procedures for 
selecting problems and for scoring the test to be used in arriving at 
the pass-fail decision. Whai: is desired is a procedure for specifying 
how a particular test should be built so that separate test construc- 
tors working independently can each construct a test that will lead 
to the same decision concerning the achievement of individual students. 

Our second area of research is concerned with getting the student 
from his initial repertoire cf behaviors to those specified in the 
training objectives. This gives rise to the problem of determining 
what other behaviors must be acquired by the student in getting from 
"here to there," and will include problems dealing with the derivation 
of mediating and transitional behaviors, hints and prompts, and various 
degrees of approximate criteria for use during each phase of training. 
The heart of this area of research, however, is the development of 
techniques for sequencing these behaviors for maximally effective and 
efficient acquisition. 

Our third area of research will be concerned with establishing 
optimal conditions of practice for each of the various types of behav- 
iors that must be acquired during training. Here we will be concerned 
with the engineering of what is knoim about learning into the training 
context* Current efforts in this area are being devoted to assessing 
and summarizing .vhat in fact is known about learning at this time. One 
major problem chat we are encountering is that of translating laboratory 
operations into applied operations. We are having to make many "best 
guesses" in order to adequately summarize the state of the learning 
science for training purpose:*; we hope to start checking on many of 
these "best guesses" in the near future in a series of small studies. 



As a preview to conducting rigorous research in these areas, we 
have been doing some informal work in automated instruction. This has 
been done primarily to acquaint the research staff with some of the 
problems involved in programming and in applying learning principles 
in the training context. Members of the staff, both past and present, 
have constructed programs dealing with electrical switches, molar 
solutions, proportions, multiplications, division, powers of 10, 
simple equations, powers and roots, nomograms, and graphs. 



Some of these programs employ alternative sequencing, although 
most of them are of the sequential type. We have also developed a 
device for teaching accurate reading of electronic meters. This 
device was developed primarily to provide our engineering staff with 
preliminary experience on the type of circuitry required by "teaching 
machines." The engineering staff is currently developing a more 
versatile device centering around a 16mm motion picture projector. 
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Plans for the near future call for the construction of a "learning 
laboratory" that will contain nine individual booths. 

Thus, our research is, and will be, concerned with three related 
types of problems: 

(1) Specifying the criterion of training. 

(2) Developing procedures for guiding and assessing learning 
during training. 

(3) Engineering known principles of learning into the 
training context. 

Instead of seeking answers to the questions "What should be taught?" 
and "How should it be taught?", we are rather asking "What should be 
learned?" and "How should it be learned?" The latter questions place 
emphasis on the behaviors that are in fact to be acquired by the student 
during training and on the procedures he is to employ in acquiring them, 
rather than on the stimulus material that is to be presented to him; 
that is, the paraphernalia of instruction in the form of textual and 
lecture material, training aids, and so forth. The specification of 
these things is secondary to the specification of the behaviors that 
must actually be acquired by the student during training. 
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THE EFFECTIVENESS .NO IMPLEMENTATION OF 
INSTRUCTIONAL CLOSED-CIRCUIT TELEVISION 

Staff Members of Division No. 5 (Air Defense) 



A means for evaluating the applicability of television 
to Army training programs is given in this paper, which 
discusses factors involved in both the construction and 
management of training programs that the training 
agency should incorporate into its courses before con- 
sidering the utilization of closed-circuit television. 



This report has been prepared in response to a request from G-3, 

U.S. Army Air Defense Center, Fort Bliss, Texas, for "a study on tele- 
vision which will provide guidance on the effectiveness, recommended 
methods of implementation, etc., as related to television and its 
application to effective instruction.” A study of this nature has 
already been conducted by the U.S. Naval Training Dtvices Center 
(NTDC) . 1 This report essentially concurs with the NTDC recommenda- 
tions and presents portions of their report. In addition, this report 
emphasizes the fact that the presentation of information constitutes 
only one of many aspects of the instructional process. These other 
aspects are listed and discussed in an Annex to this< report. 

Whether the employment of instructional closed-circuit television 
will improve the effectiveness of training is primarily a matter of 
the adequacy of the training against which it is compared. Closed- 
circuit television is not a remedy for all training ills. It is 
primarily a technique for making certain types of instructional material 
simultaneuusly available to large numbers of students . The Pennsylvania 
State University television research project, 2 for instance, concludes 
that: "A break-even point between conventional instruction (in groups 

of 45) and televised instruction was estimated to bs about 200 students 
This break-even point is based on the use of vidicoi equipment which is 
less expensive than orthicon equipment. j 



1 Instructional Television Research Reports , Technical Report No. 
NAVTRADEVECEN 20-TV- 4, NAVEXOS P-1544, U.S. Naval Training Devices 
Center, Fort Washington, L.I., N.Y., June 1956. 1 

2 C.R. Carpenter, arid L.P. Greenhill. Instructional Television 
Research , REPORT NUMBER WO: Ihe Academic Years 1955-1956 and 1956- 

195?, Ay\ Investigation of Closed Circuit Television for Teaching 
University Courses j Division of Academic Research and Services, 
Pennsylvania State University, University Park, Pernsylvania, 

Spring 1958. 
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The Naval Training Devices Center has prepared a checklist’ (Fig- 
ure 1) for evaluating the applicability of television to Army training 
programs (Figures are in a group starting on page 22 of this paper.) 
Their report recommends that the checklist be used by a committee of 
three instructors in a local situation, and that these instructors be 

familiar with the training schedules of the organization concerned and 

know thoroughly the training situation of that organization vath all 
its special problems. Tn using this checklist it is not necessary for 
the evaluators to be aware of the fact that their evaluation may 

determine whether or not a given course of instruction is to be pre- 

sented via television, Before television should be considered as an 
alternative presentation media, however, the training agency should 
ensure that it has done everything possible to make its training pro- 
grams maximally effective and efficient within their current procedures. 
When this is done, it nay frequently be found that the original problem 
for which television vas considered no longer exists. 

An Annex to this report presents a listing and discussion of fac- 
tors involved in both the construction and the management 6f training 
programs which a training agency should incorporate into its programs 
before considering the utilization of instructional closed-circuit 
television. A careful detailed analysis on the part of the training 
agency may indicate that its difficulties in training, including the 
existence of a large student population, can be solved more effectively 
by techniques other than closed-circuit television. For instance, a 
detailed analysis of the job requirements may show that a large amount 
of irrelevant material can be pared out of its training programs; this 
will shorten the training programs and consequently decrease the number 
of students per instructor. Perhaps improvement of the effectiveness 
of training will require more opportunity for the students to apply 
and practice the required skills and knowledges; in this case, closed- 
circuit television would be of no value. Analysis may show that the 
existing training programs cannot be determined to be effective because 
of inadequate evaluation procedures, or that its examinations either 
are not administered frequently enough or are not sufficiently realistic 
to pinpoint training problems. 

The NTDC checklist is reproduced in Figure 1. Tneir report states: 

11 J n general, the greater the number of marks after (lj, the 
greater the adaptability of the lesson to instruction by television. 
Marks after (2) either means that the lesson contains no advantage 
for television, or that presentation by television may even in some 
cases be contra-indicated The criteria checklist is a general 

guide in early stages of experience of television as an instruc- 
tional medium. Fxom a list of subjects those wJth a high (1) count 
(favorable for TV) can be quickly selected and considered for tele- 
vision use, but the final decision to teitvise a lesson or not 
should be made by the local command on the basis of the ltcal 
situation. Favorable rating of a lesson on the criteria checklist 
is not a mandate to televise but rather indicates feasibility". 
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Some of the items in this checklist deserve comment in addition 
to that contained in the NTDC report. 

Item 1, Type of instruction . If instruction is being conducted 
in accordance with the Army Field Manual ^:l-6 it may be difficult to 
find lessons that are purely presentation and/or demonstration. In 
discussing the application of the stages cf instruction FM 21-6 states: 

: "The stages of instruction serve as a checklist for the instruc- 

tor in choosing teaching procedures. Whenever practical he applies 
all 5 stages to each lesson presented It is often better to 
present certain subjects in small segments processing each segment 
through all the stages of instruction including the application 
and examination stages before going to the next segment. . , . 

In controlled practice, when material is presented step-by-step, 
the presentation and application stages are' combined . Flexibility 
is the key to successful use of the stages of instruction". . , , 
"However, the instructor must study every instructional situation 
for opportunities to secure student participation in the applica- 
tion, examination and critique stages. He must strive for the 
complete teaching process in which he can plan, tell, show, do, 
check and review and/or critique". 

Item 6, Supply of Training Aids . Currently used tvaining aids 
may be in short supply because of their cost and/or complexity. Care- 
ful consideration should be given to whether or not such aids can be 
simplified and made less costly. Some evaluation should also be made 
a: H » to their effectiveness in training. Only those training aids which 
a:::e in short supply, whose training effectiveness and efficiency have 
b-;en demonstrated, and which cannot oe made either less costly or less 
complex should be considered in deciding whether or not a given lesson 
should or should not be given via closed circuit television. Figure 2 
gives a list of training aids. 

Item 7. Portability of Training Aids. The portability of training 
aids is not a problem if students can be readily brought to the aid 
rather than the aid to the students. The definition of "readily," 
however, will vary from one specific instance to another depending upon 
the mode and availability of transportation, weather, number of students 
par class, parking facilities, and so forth. 

Item 8. Viewing Training Aids . Currently used aids may not always 
be the most appropriate. Many graphic aids, for instance, could easily 
be reproduced in student manuals, thus providing the student with what- 
ever detail he may need and making the aid available to him for 
out-of-class study. 

One other rationale frequently given for employing instructional 
closed-circuit TV is that it makes the best instructors available 
to a greater number of students. An extensive study conducted by the 
Air Force Personnel and Training Research Center, 1 however, indicates 



*J , E. Morsh, and E.W, Wilder. Identifying the Effective Inst'i'uotov : 
A Review of the Quantitative Studies , 1900-1952, Air Force Personnel and 
Training Research Center, Lackland Air Force Base, San Antonio, Texas. 
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that there are no reliable and valid techniques currently available for 
identifying effective instructors, where effectiveness is defined in 
terms of student learning. 

In addition, it is interesting to note that in the Pennsylvania 
State University studies a hypothesis for future experimentation is 
formulated to the effect that methods of instruction arc relatively unim- 
portant, They are, however, using the phrase ''methods of instruction" 
to mean "methods of presentation," Instruction would also include such 
factors as application and practice on the part of the student, and 
examination and review. This report points out that "two , . . potent 
factors are likely to be (a) repetition of the presentation of important 
concepts and (b) logical progression from one topic to the next." 

If a training agency does decide to install instructional closed- 
circuit television equipment (i.e,, if it finds that it has 200 or nore 
students at the same point in training at a given time in a relatively 
substantial proportion of its courses; if it finds that its overloaded 
courses contain a substantial proportion of whole clarsbs which are of 
a presentation and/or demonstration type, and if it finds that it has 
a shortage of instructors, equipment or facilities), then it should 
give careful consideration to how it utilizes its television equipment, 
The NTDC report lists some principles to guide the utilization of 
instructional television, These are reproduced in Figure 3, 

Item 1 states the m<2xii7rm number of studt ts that should be allowed 
to view a single receiver. There may be some appli .tions, however, in 
which this number shouli be reduced. Item 6 in this list has already 
been commented upon. With respect to Item 8 on the use of an inter- 
communicating system, it should be noted that the Pennsylvania State; 
University studies found no advantage in such use. The Annex contains 
some further amplification of this point. The studio panel mentioned 
in Item 7 is strictly a matter of instructor choice. In several studies 
it has been indicated that instructors frequently prefer to have a 
studio panel available during their early television experiences, but 
that as the instructor becomes more adept and proficient in his tele- 
vision presentations, ho generally no longer feels the need for a 
studio panel. The Annex contains some further guidance as to the use 
of in-course achievement tests which are mentioned in Item 9 of Figure 3, 
In a training situation achievement tests should be based not upon the 
curricula but rather upon the objectives of the training program. 

Figure 2 presents the NTDC recommendations with respect to the use 
of training aids in instructional television* In constructing training 
aids to be used with black and white television, primary consideration 
sliOuld be given to the brightness rather than the color characteristics 
of the aid, since it is the brightness characteristics which will be 
reproduced at the receiver. 

Figure 4 presents the NTDC recommendations with respect to tele- 
vision equipment. 

Figure 5 gives the NTDC recommendations with respect to television 
operational techniques. With regard to Item 1 in this list, in those 
instances in which students are expected to take notes or to copy 



diagrams, drawings, outlines, and so forth, care should be taken to 
ensure that the material is kept on camera long enough for the student 
to do so and that other material not be presented during this time, 
that is, the instructor not continue lecturing during these pauses in 
the television presentation. In general, effective television instruc- 
tional practices differ considerably from commercial practices. The 
instructional presentation should specifically avoid unnecessary or 
irrelevant camera motion, dramatic effects, unusual lighting, and in 
general the "slickness” associated with commercial productions. These 
things do not add to the instructional value of the presentation. It 
is much more important for instructional purposes to present the appro- 
priate information in an instructional ly effective manner and sequence 
and in a style characterized primarily by its clarity and containing 
as few distracting characteristics as possible. 



SUMMARY AND CONCLUSIONS 

This report, prepared in response to a request from G-3, U.S. Army 
Air Defense Center, Fort Bliss, Texas, for "l study on television v:hich 
will provide guidance on the effectiveness, recommended methods of 
implementation, etc., as related to television and its application to 
effective instruction, 11 essentially concurs with a study conducted by 
the U.S. Naval Training Devices Center. 

The following conclusions are presented: 

(1) Closed circuit television is not a remedy for all training 
ills. It is primarily a technique for making certain types of instruc- 
tional material simultaneously available to large numbers of students. 

(2) The checklist shown in Figure 1 is recommended for evalu- 
ating the applicabil ly oi television to Army training programs. 

(3) Before considering television as a means of presenting 
material, other factors affecting the value of training programs should 
be considei d. The Annex presents a listing and discussion of factors 
involved in both the construction and management of training programs 
which the training agency should incorporate into its courses before 
considering the utilization of closed circuit television. 

(4) Principles to guide the use of instructional television 
are presented in Figixe 2. 

(5) NTDC recommendations with respect to the use of training 
aids with television are presented in Figure 3. 

(6) Figure 4 presents NTDC recommendations concerning tele- 
vision equipment. 

(7) Figure S presents NTDC recommendations concerning tele- 
vision operational techniques. 



Checklist for 

Evaluation of Army Training Subjects 



Lesson Date 

(write in if not already filled) 

Make a check mark for every item below according to your information < r 
best judgment. 

Think of the lesson in terms of a specific 50-minute period. 

1* Type of Instruction 

(1) Presentation and/or demonstration. . (2) Application . . 

(Mostly observation by trainee) (Training by doing) 

2. Instructor-student intercommunication (two-way conversation) 

(1) Not necessary (2) Necessary . . . 

3. Rapid dissemination of information (reaching many men early in the 

training or rapidly distributing new information) 

(i) Necessary (urgent) (2) Not urgent. . . 

4. Potential supply of instructors due to difficulty of subject matter 

(difficult, usually a shortage; easy, many available) 

(1) Pew 12 ) Many 

5. Physical risks (danger in the training situation) 

(1) Great __ ( 2 ) Little 

6. Supply of training aids 

(1) Short (more needed) (2) Sufficient or 

plentiful . . . 

7. Portability of training aids 

(1) Difficult (2) Easily moved. . 

'large, heavy, unwieldy) (light, easily 

handled) 

8. Viewing training aids 

(1) Close-up necessary (2) Ciose-up not 

necessary . . . 

9. Color used in training aids 

(2) Color essential for teaching . . . . (1) Not needed, un- 

important * . . 

10. Making a sound-film record 

(1) Highly desirable (2) Not desired . . 

11. Training time lost, moving from area to area 

(1) Much (2) Little 

32. Does weather interfere vith instruction? 

(1) Yes (2) No 

13. Security classification? 

(1) Unclassified or restricted (2) Confidential, secret, 

top secret. . * 



Figure 1 
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NTDC Recommendations With Respect to 
the Use of Television Training Aids 



It is recommended that: 

1. Fine-lined schematics of letter size, &h x 11, be televised 
in the usual printing of black on white instead of 
reverse typography. 

2. Large charts be made in contrasting shades of gray; gray on 
medium-brown background if possible. 

3. Flannel-graphs, magnetic boards, and sectional flip-cards be 
used to economize tir.e and to increase trainee attentiveness. 

4. Soft, white chalk be used on a blackboard; yellow chalk on a 
green board given preference when available. 

5. Models, mock-ups, cut-aways, and training boards to be us>^d but 
that original equipment be substituted whenever possible because 
magnification can be produced by television. 

6. Attention be given to colors and gray-values; that re-painting 
be done where shown to be needed by actual tiy-out. 



Figure 2 



Utilization of Instructional Television 
ir. Army Teaching 



It is recommended that; 

1. The number of students viewing one television receiver screen 
be limited to 20. 

2. Television instruction be limited to four hours per day; each 
continuous viewing session to not more than 30 minutes; and 
that there be provision in the schedule for intensive dis- 
cussion daily. 

3. Television subjects be taught as an integral part of the cur- 
riculum with changes kept :o a minimum. 

4. Instruction be informal and extemporaneous. 

5. The training situation be instructor-dominated. 

6. The best available instructor be used for television. 

7. A panel of three men be used in the studio if a partial sub- 
stitute for class discussion is desired. 

8. Each local training command decide whether or not to use an 
intercommunicating system. 

9. There be achievement testing of the type usually employed. 

10. Instructors thoroughly familiar with their subject matter and 
experienced in teaching procedures be selected for television 
end given some practice "dry runs". 

11. Actual demonstrations supported by visual aids be presented 
on television. 

12. Attitud ; nal subject matter as well as technical be televised. 

13. The field use of television be given experimental consideration. 



Figure 3 



NTDC Recommendations Concerning 
Television Equipment 



It is recommended that: 

1* The inexpensive but effective lapel microphone be used. 

2. Two cameras be provided in the studio but that it be kept in 
mind a satisfactory job of teaching can be done with one 
camera in case the other has failed. 

3. A large studio monitor, properly placed, be provided so that 
the instructor can readily observe his own teaching. 

4. Fluorescent lighting be used in the studio with portable flood- 
lights on training aids when needed. 

5. Lighting in classrooms be practically normal, avoiding reflec- 
tions by the television receiver screen. 

6. In mass teaching, inter communioat ion be provided inly where 
deemed indispensable, to be determined by local command. 

7. A large size television receiver screen, 19 inches or larger, 
be used in the classroom. 

8. Large screen television projectors not be used. 

9. Simple, inexpensive tripod camera dollies be used. 

10. Distractions due to ,r burning-in" effects on tubes be avoided 
by proper manipulation of studio equipment. 

11. Proper card stands, holders and other display devices 
be provided. 

12. Television projection equipment be provided for the showing of 
films, film strips, slides and kinescope recordings. 

13. The possibility of producing kinescope recordings be given 
serious consideration. 

14. A closed circuit be considered standard but a supplementary 
open-circuit added where local requirements are deemed to make 
it highly desirable. 



Figure 4 



NTDC Recommendations Concerning 
Television Operational Techniques 



It is recommended that: 

1. Diagrams and drawings be kept compact on a blackboard, that 
the amount of writing be somewhat restricted, and the rate of 
writing deliberate. 

2. Close-up shots be used liberally. 

3. Super-impositions be used where it is desired to show 
relationships . 

4. Fades and dissolves be employed as transitional techniques 
but that they should be used sparingly. 

5. An occasional panning of the camera be used for spatial orien- 
tation and t:> provide a panoramic effect. 

6. Whenever a subjective presentation is an advantage thai over- 
the-shculder (zero angle) shots be used. 

7. Studio settings be simple (' quiet") to avoid the distractions 
of a complex ("noisy") setting. 

S. A lesson be instructor-dominated and remain essentially so when 
two cameras are used. 



Figure 5 
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ANNEX 



Factors To Be Considered in Training Programs 



The instructional process consists of two basic facets: 

(1) The presentation of information to the student by an 
instructor or instructional device, 

(2) The learning of new knowledges and skills by the student. 

in constructing and adml .us tering a training program, emphasis 
should be put upon the learning of knowledges and skills by the student 
which should, in turn, determine the types of informational presenta- 
tion to be made to the student. The first step in the construction of 
any training program is that of specifying the knowledges and skills 
that the student will be required to learn as determined by the require- 
ments for the job for which training is being instituted. The second 
step is that of determining the sequence in which these knowledges and 
skills may be most efficiently learned. Learning invariably requires 
practice of that which is to be learned. Thus, the? third step is that 
of determining the optimum conditions of practice for each of the 
knowledges and skills which the student must learn. The fourth step 
is that of determining the optimum type of informational presentation 
to make to the student in order to obtain the desired learning of 
knowledges and skills in the appropriate sequence under optimum con- 
ditions of practice. 

Flexibility should be a part of every training program. Even though 
students are relatively homogeneous with respect to educational back- 
ground and relevant aptitudes, these indices are not sufficiently pre- 
cise to rule out all relevant differences between students or groups 
of students. Deviations from the initial instructional plan should be 
provided for and should be based upon the actual amount and quality of 
learning encompassed by tie student. 

In order to effectively implement the instructional process, the 
training program should include the following operational techniques 
and procedures: 

1. The Specification of Training Objectives 

The first problem which must be resolved by a training agency 
is that of deciding which knowledges and skills must, in fact, be learned 
by a student in order to perform adequately on the job. The solution to 
this problem must invariably be based upon a detailed analysis of the 
performances actually required by the job; i.e., a detailed task analy- 
sis. This is probably the rr.Dst crucial step in the construction of any 
training program since errors in this step quickly become compounded in 
all other steps. 

If relevant skills and knowledges are excluded from the 
statement of training objectives, training cannot possibly produce 
adequate job performance. If a substantial proportion of irrelevant 
skills and knowledges are included in the training objectives, job 
performance may be degraded and the cost of training will certainly be 
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unnecessarily high. The exclusion of relevant skills and knowledges 
is probably the more damaging alternative. There is little point in 
expending training time without turning out a usable product. 

Responsible training agencies must invariably be faced 
with che problem of deciding at the termination of training which 
students have, in fact, learned the knowledges and skills specified 
by the training objectives and which have not. The results of the 
major examinations employed during training and at its termination, 
constitute the basis upon which this decision is based. Thus the 
statement of training objectives not only specifies the knowledges 
and skills which the student must learn during training but also pro- 
vides a basis for the construction of achievement tests for determining 
whether or not such learning has actually taken place. Achievement 
tests should be bui.t specifically to assess the stated objectives of 
training. The items in such tests should be selected on the basis of 
their congruity with the knowledges and skills specified in the state- 
ment of training objectives rather than on the basis of their difficulty 
or homogeneity for any given population of students. 

The statistical characteristics of test items are largely a 
function of the quality and character of the instruction. Ideally suc- 
cessful training programs should produce students, all of whom make the 
maximum possible score on the program’s examinations. This, of course, 
is not possible in the practical situation but it should be maintained 
by trainers as a model towards which to work. Instruction is effective 
only to the extent that appropriate learning occurs. 

2. Sequencing * 

The knowledges and skills which the student is required to 
learn should be sequenced in such a manner that for any given point in 
training the student will have been provided with all the necessary 
background information (i.e., knowledges and skills) in the preceding 
phases of training. In order to make the student's learning more 
meaningful to him at each point in training and to maximally facilitate 
the integration of the knowledges and skills he is required to learn, 
the sequence in which he learns these knowledges and skills should 
generally follow a whole-to-part order. In addition, the skills and 
knowledges should be sequenced so as to discourage erroneous learning. 
Erroneous learning is wasteful of training time. 

3. Conditions uf Practice and Feedback 

In order for learning to occur, that whK i is to be learned 
must be practiced. Passive receiving of information on the part of the 
student is inadequate for maximally effective learning. Much of the 
student's practice may be ir. terms of "thinking" during lecture presen- 
tations. In addition, much of it may occur during out-of-class study. 
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Neither of these kinds of practice, however, can be readily 
observed by the instructor. Thus, he cannot be assured that his students 
are, in fact, learning skills and knowledges required by the training 
objectives. Consequently, if he does not frequently check upon the 
progress of his students, he may fail to correct the defects in his 
instructional presentations until it is administratively too late to 
take appropriate remedial action. He may check upon student progress 
by allowing his students a brief period during which they may ask ques- 
tions, by discussion, by his asking questions of a few students or by 
administering short quizzes during each instructional period. 

Free questioning or undirected discussions on the part of 
the students as a primary basis for remedial action tends to be too 
inefficient and places too much responsibility upon the student for 
knowing what constitutes a relevant question. Spot checking of a few 
students may not always give an accurate picture of the achievement of 
the whole class, particular ly if t lie class is larje. Brief quizzes 
avoid both of these difficulties. The instructor is in a much better 
position to know what specific points in a presentation ate of greatest 
relevance than are the students, and a quiz will provide him with 
information about the achievement of all his students. 

The instructor has some latitude in selecting the point in 
training at which he chooses to evaluate the effectiveness of a partic- 
ular hour or topical unit on instruction. He may give a short quiz 
either at the end of an hour of instruction or at the beginning of the 
next hour of instruction, which will generally be on the following day. 
Purely can an instructor expect his students to learn the skills and 
knowledges specified for an hour of instruction purely as a function 
of his instructional presentation alone. Thus, if he quizzes at the 
end of the hour of instruction, he should restrict his items to those 
assessing skills and knowledges which the student might reasonably be 
expected to learn as a function of that hour of instruction without any 
additional practice. This procedure provides him with more time in 
which to consider the type of remedial action he night take if the 
results of the quiz indicate that adequate learning has not occurred. 

If he quizzes at the beginning of the following hour of 
instruction, he can probably employ more adequate test items since he 
will be able to assume some practice beyond that given during the 
instruction itself. This procedure, however, requires that the 
instructor be informed about the student r s performance on the quiz 
within a matter of minutes and that he can decide what remedial action 
is necessary and put it into effect immediately. It should be noted 
that any training program which takes a need for remedial action during 
training into account must also provide for instructor continuity over 
at least several successive hours of instruction. Otherwise, the 
instructor would have no opportunity in which to take the indicated 
remedial action. 

Since* a gnat deal of the student's practice may occur 
outside of the classroom, the instructor should also employ techniques 
for preventing erroneous learning and practice when the student is out 
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of his direct control. He may attempt to do this by urging his students 
to take careful notes during his lectures and use the notes as a means 
of guiding their out-of-r. lass study. He would be hard pressed, however, 
to actually insure that the students' note-taking in itself was accurate 

There are two practical alternatives open to him. First, he 
might write crucial information on the blackboard as he lectures. Since 
he will probably also want his students to carefully follow his lecture 
presentation as he gives it, and since note-taking frequently interferes 
uith the students 1 attending, the instructor should pause sufficiently 
often to allow the student to copy the material on the blackboard and 
thus reduce the interference which might otherwise arise as a result of 
the student's attempting to do two things at the same time (i.e., attend 
to the lecture presentation and take notes). Secondly, the instructor 
may give each student a reproduced set of pre-prepared lecture notes 
and thus completely do away with the necessity of the students 1 taking 
any notes at all. 

4. Methods of Presentation 

The instructor who does not employ techniques for finding 
out what knowledges and Skills the students are in fact learning as a 
function of hi s instructional presentation is not teaching but only 
submitting his students to auditory or visual bombast. In comparing 
one method of instructional presentation with another, it is necessary 
to take into consideration the practice conditions used in conjunction 
with earn method of presentation. As one learning theorist has pointed 
out: 1 "A student does not learn what was in a lectui'e or book. He 
learns only what a lecture or book causes nim to doV. 

Thus, the problem of selecting a method for presenting 
instructional information is essentially one of selecting a presenta- 
tion which will induce the student to "do" those things which he is to 
learn. This is true not only in selecting a general method of presen- 
tation for a particular unit of instruction, but also in selecting 
training aids and devices. Emphasis should be on selecting aids and 
devices which induce the student to "do" those things v.hich he is to 
learn. For instance, if a student is working with a special skill 
trainer, the trainer needs to resemble the equipment which the student 
will iatt* use on the job only to the extent that it will induce the 
same response from the student as does the actual equipment. 

Care should be taken to insure that informational presen- 
tations are not too long or complex to provide adequate control over 
the students 1 practice and application. Information in the initial 
phases of a long, complex presentation may well be forgotten by the 
student before the presentation is complete. For this reason Field 
Manual 21-6 recommends that . . it is often better to present 
certain subjects in small segments, processing each segment through 
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all the stages of instruction, lncludirg the application and examination 
stages, before going to the next segmert. M 

Care should also be taken to ensure that the method of 
presentation selected for a particular phase of instruction can in 
fact be implemented within the local training situation. For instance, 
the so-called "conference method" specifically attempts to establish 
classroom interchange between student and instructor. But it requires 
an instructor of such rare competence, experience, and talent as to 
be impractical for large training efforts. It almost invariably 
deteriorates into ordinary lecture presentations. 



o 

ERIC 



31 



35 



AUTOMATED INSTRUCTIONAL METHODS 
FOR TECHNICAL TRAINING 1 

Paul G. Whitmore 



This paper explains attempts to obtain a basis on which 
to estimate the scope of applicability of automated 
instructional techniques to Army technical training. 
Experiments, training procedures, and problems are 
described and illustrated. 



The training of men for combat effectiveness has traditionally been 
one of the major functions performed by an army, An arm, is not simply 
an accumulation of men, but rather an accumulation of men possessing 
knowledges and skills pertinent to the organization's combat effective- 
ness. The number and complexity of these knowledges and skills and 
their availability among current and potential U.S, Army personnel ale 
the main factors determining the magnitude and scope of the Army’s 
training effort . 

The development of complex weapon systems has increased and is con- 
tinuin'* to increase the knowledge and skill requirements which must be 
net by Army personnel. Men possessing appropriate technical knowledges 
and skills are not available in sufficient numbers in the general popu- 
lation. Therefore, the Army must conduct an extensive training effort 
in order to obtain technically qualified personnel. Such training is 
both costly and time consuming. An ever-increasing portion of soldier 
enlistments is being consumed by this necessary training, thus leaving 
less time available for the conduct of military duties. Consequently, 
it is necessary that the Army be provided with more efficient training 
procedures in order to arrest, and possibly even reverse, this trend. 

The objective of research under Work Unit TEX7RUCT is to develop 
more efficient and effective instructional methods applicable *o Army 
technical training. The first part of the research was devoted to the 
conduct of literature reviews and informal studies as a means of 
identifying and defining the research problems for future research. 

Early efforts were concerned with exploring methods for improving 
group instruction. It was felt that the efficiency of group instruc- 
tion might be improved by providing the instructor with means for 
obtaining information from his students concerning the degree to 
which they were able to understand all of his presentation while he 



briefing developed for presentation in December 1960. 




32 




w?,s lecturing. A simple device was developed for obtaining and present- 
ing this information. It consisted of a hospital bell-push switch of 
each of 10 response stations, a linear DC amplifier, and a sumnating 
meter. Several lectures on a variety of subjects were presented to 
audiences of unit personnel. Audience responses to these lectures were 
recorded on a moving paper tape. The audience members were requested 
to depress their bell-push switches whenever the lecturer’s presenta- 
tion became difficult to follow or understand. The lecturers agreed 
that for the most part audience response came too late for them to know 
with sufficient precision what part of the lecture had not been ade- 
quately understood. Audience reaction was simply not fast enough. 

Studies conducted at other laboratories using similar devices have 
found that instructors generally consider such devices to be too cumber- 
some for effective use. Certainly the ability to instantaneously vary 
a lecture presentation in response to audience reaction requires con- 
siderable skill and mastery on the part of the instructor. Additional 
studies of group instructional methods were not conducted since it was 
the opinion of the research staff at this time that large gains in 
instructional effectiveness and efficiency were not to be made in the 
group situation. 

The decision to abandon the group situation and instead concen- 
trate almost exclusively on the individual situation was largely 
predicated on the highly promising results obtained by researchers at 
other laboratories with automated instruction or "teaching machines." 

For instance, in one study a reading capability of German equivalent 
to one semester of college work was obtained in a mean total tine of 
47^ hours. In another study a one-semester college algebra course 
was taught in a mean total time of 30 hours. A conservative estimate 
would indicate that college stt dents spend at least 75 hours total 
time in a one- semester course taught by traditional methods. Thus, 
in these two instances the use of automated techniques led to a savings 
in time of from one-third to better than one-half. 



In a third study in which automated instructional techniques were 
applied to an introductory college psychology course, the median score 
on the final examination was 94%, and r.o student scored less than 85%. 
Previously, using traditional instruction methods only the very rare 
student scored as high as 85% on the examination. Automated techniques 
have beer used for teaching second- and sixth-grade spelling, second- 
grade arithmetic, introductory college statistics, binary arithmetic, 
English grammar, introductory college physics, symbolic logic, and 
electronics troubleshooting. Thus, it would appear that these tech- 
niques not Oily offer great promise but also have a wide range of 
appl icabi 1 i f y . 

Automate i instruction is primarily concerned with engineering into 
the instructional situation four important learning principles which 
are not generally accounted for by traditional instructional methods. 
These four principles are: 



(1) Continuous participation by tbe student in the instruc- 
tional process. 
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(2) Providing immediate knowledge ox results to the student 
for each response that he makes. 

(3) Recognition of individual differences in rate of learning. 

(4) Providing a high rate of success for the student through- 
out learning. 

Let us consider how these principles are actually applied in the 
construction of an automated program. For example, in one of a series 
of programs each page can present the student with a small amount of 
information and require that che student in turn respond to each page. 
In this way the program requires continuous participation on the part 
of the student. On the back of each page is the correct answer which 
the student should have given, thus providing him with immediate 
knowledge of results. Since the program provides individual instruc- 
tion, it allows each student to proceed at his own rate and thus 
accounts for individual differences in rate of learning. 

The most difficult principle to implement is that of providing 
the student with a high rate of reinforcement or success throughout 
learning. Ideally, we would like to have programs by which students 
could learn without making a single mistake. To build a program that 
even approximates this ideal requires intensive analysis of material 
to be learned, plus several carefully conducted tryouts of provisional 
programs. The primary problems are concerned with appropriately frac- 
tionating or segmenting and sequencing the material. Basically two 
different methods of sequencing have been proposed. B.F. Skinner of 
Harvard University introduced the sequential method in which every 
student proceeds by the same route through every step in the program. 

He is more or less forced to employ sequential programming by his 
theoretical bias favoring constructed responses. Norman Crowder of 
Western Design, Inc., developed the alternative method of programming 
in which different students proceed by different routes through vary- 
ing steps. This flexibility is made possible as a consequence of 
employing response selection instead of response construction; that is, 
the alternative method employs mu ltiple-chcice type items in each step. 
The alternative method shows promise in being able to account not only 
for individual differences in rate cf learning, but also for differ- 
ences in aptitudes and in previous achievement. 

In order to become familiar with the problems involved in building 
these types of instructional programs, members of the research staff 
built several on various topics which were presented in book form. 

The first two of these pregrams— molar solutions and electrical 
switches — were not intended for operational use. However, at the 
request of the Air Defense School, the staff built a series of nine 
programs to be used by applicants to the Air Defense Missile Officer 
Basic Course as a means of correcting deficiencies in mathematics 
knowledge prerequisite to the course. These programs provide instruc- 
tion on multiplication and division of decimals, cancellation, powers 
and roots, povers of 10, logarithms, proport ions , stated algebraic 
problems, reading nomograms, and reading graphs. These programs have 
been informally administered to a variety of individuals including Army 
officers and enlisted men, civilian employees, and college students. 
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In addition, a college freshman algebra program developed by Gilbert 
at another laboratory was administered to a few teen-age and near-teen- 
age children of the staff and some of their neighbors. Since these pro- 
grams are all concerned with verbal and symbolic material, an experi- 
mental device for teaching a perceptual skill was also constructed and 
tested. This device was designed to teach the ability to read a meter, 
but proved to be a dismal failure for its intended use. However, it 
did teach the research staff a considerable amount about the require- 
ments for designing and constructing such devices. Although these 
experiments were extremely informal, they did provide the kind of ini- 
tial guidance necessary for the planning of a research program. 

It readily became apparent that because of the precise control over 
the instructional process afforded by these automated techniques, it 
is necessary to specify the objectives and achievement prerequisites 
of an automated program in minute detail before a program can be built. 
This difficulty was most pronounced during the construction of the 
graph reading program and was probably due to the fact that graph read- 
ing is usually not taught as such or is not taught at, all. Consequently, 
there was no cere of implicit detailed objectives available to the pro- 
gram writer. Although a set of general objectives had been prepared in 
advance, they were found to provide an inadequate basis for the many 
decisions required during the construction of the program. What orig- 
inally appeared to be an adequately stated objective was found still 
to contain too many alternatives from which the program writer might 
choose. To have included all possible alternative objectives would 
have resulted in a large, unwieldy, unusable program and would have 
required too much time for its development. 

The segmentation or fractionation of the course content deemed the 
most rational or logical to the program writer or subject matter expert 
may not, and probably will not, be the most adequate for instructing 
naive students. In obtaining the coordinates of a point on a graph, 
for instance, instruction should teach the student to move from the 
given point to the axes, rather than fro*., the origin along the axes 
to the point; that is, it is necessary to distinguish between the act 
of reading a graph and the act of constructing a graph. In preparing 
the program on molar solutions, it was found necessary to distinguish 
the act of determining the amount of solute present in a solution of 
known molarity from the act of determining the amount of solute needed 
in order to prepare a solution of given molarity; that is, it was 
necessary to distinguish the act of analysis from the act of syntheais. 

A subject matter expert may frequently fail to make such fine distinc- 
tions between the var.ous directions in which relationships may exist 
or may fail to adequately identify the behavioral elements of an act 
relevant to its instruction. 



In several instances, students complained that t lie automated 
material was too easy, although the same material presented by tradi- 
tional methods has been among the more difficult topics in the course. 
One of the children to whom Gilbert’s algebra program was administered 
easily completed the first two lessons successfully then lost interest 
in the program. Several of the college students who tried the program 
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on molar solutions complained of its being too easy, although they 
did complete the program. Dr. Jesse Day, chairman of the Chemistry 
Department of: Ohio University, tried the molar solution program in 
some of his freshman classes and was'so gratified with the results 
that he programmed the kinetic theory of gases for juniors in his 
physical chemistry class. Again his students learned the material 
better than they ever had by conventional methods, but this time he 
received severe complaints about the program’s being too easy. Ques- 
tioning brought out the fact that the wives of some of the students 
had worked through their husbands’ programs, learned the material, and 
were unable to understand why their husbands complained about the 
difficulty of the course. 

It further appears that even with these techniques, special atten- 
tion must be given to the problem of obtaining adequate retention over 
long periods of time. It is possible for students to work through 
these programs successfully and still not retain the material any 
better than they would by conventional methods. Immediate retention 
within the context of the program may carry a student through the pro- 
gram with little or no difficulty, but yet not insure the permanency 
of the learning. This is a problem easily overlooked in program con- 
struction and essentially requires that long-range retention be handled 
as a problem separate from immediate understanding. 

This first effort has identified certain researchable problems con- 
cc :ned with the derivation and specification of training objectives and 
prerequisites, with the fractionation and sequencing of training content 
and with the retention of learned material over time. One of the major 
problems in the area of training, however, is that of relating the many 
factors which go into the instruction and administration of a course 
into some coherent whole. Since the interrelationship of these factors 
has not been established at this time, it is not possible to begin 
separate research projects on each. Rather, it is first necessary to 
generate an approximate solution for the whole to provide guidance con- 
cerning their interrelationship. 

Problems that need to be explored relatively soon are those incident 
to the construction and management of a fairly long, heterogeneous, 
largely automated instructional program. It is necessary to obtain 
some realistic basis on which to estimate the scope of applicability 
of automated instructional techniques to Army technical training and on 
which to estimate the general organizational and administrative policies 
that may be required in order to effectively implement automated instruc 
tional techniques on a large scale. 

Tentative answers are needed to such questions as: 

(1) r >fiat kind and how nuny of the Army’s training requirements 
are amenable to automated instructional techniques? 

(2) What kind of instructors or supervisors would be required 
for partially or fully automated courses? 

(3) Can purely automated courses maintain active student 
interest over long periods of time? 
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(4) How can automated courses which emit students individually 
rather than in groups be meshed together into an integrated 
training program? 

One vehicle which appears attractive as a context for such develop- 
mental research is basic electronics in which the content of the course 
consists of the skills and knowledges common to many specific electron- 
ics maintenance positions. 

There are several reasons for choosing basic electronics as the 
content vehicle for such research. First, it would provide the heter- 
ogeneous content necessary for estimating the scope of applicability of 
automated instructional techniques to Army technical training. Second, 
such a course would be sufficiently long (at least several weeks] to 
entail most, if not all, of the construction and administrative prob- 
lems that might arise in attempts to implement automated instructional 
techniques on an Army-wide basis, but not so long as to be prohibitive 
in terms of the amount of time that would be required to automate it. 
And, third, the development of an automated basic electronics course 
would be of direct value to the Army in that this specific content is 
important for so many Army jobs. 

Certainly optimal solutions to all of the problems encountered 
during the course of such research could not be generated. However, 
even though the solutions which may be generated may not be optimal, 
they would at least not leave the problems completely unresolved and 
many of them would be quite applicable to training programs utilizing 
conventional instructional methods. Refinement of these solutions 
would in turn become the subject of future research efforts. 

In order to meet these research requirements, a second project 
has been proposed and approved to determine the feasibility of auto- 
mating a basic electronics course for use by the Air Defense School. 
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DERIVING AND SPECIFYING INSTRUCTIONAL OBJECTIVES 1 



P.G. Whitmore 

This paper stresses the necessity for statements of 
instructional objectives in the construction of mass 
automated teaching programs, and also the need to 
develop rationales and procedures for contriving ter- 
minal behavior patterns. The efficiency of instruc- 
tional control is also determined by the behavior 
capabilities of the student prior to instruction. An 
example of a verbal hierarchy is given. 

GENERAL STEPS INVOLVED IN CONSTRUCTING MASS INSTRUCTIONAL PROGRAMS 1 

Let us look at a student in an instructional situation. We see a 
human organism engaged in the sporadic production of relatively discrete 
behaviors, as, for instance, practicing a perceptual-motor skill, read- 
ing a text, or writing, If we were actually to become the student 
(i.e., if we were to shift frcm external to introspective observation), 
we would become aware of a continuous stream of activity, consisting of 
both explicit and implicit behaviors. Instruction is concerned with 
the design and arrangement of situations to control this introspec- 
tively observable stream of behavior so as to modify it to conform to 
some predetermined pattern. 

There are two things that need to be determined before such situa- 
tions can be designed and arranged: 

1. The behavior pattern the student should be capable of performing 
at the end of instruction. 

2. The behavior capabilities possessed by the student before 
instruction begins that are relevant to the terminal behav- 
ior pattern, 

These two behavior capabilities define the behavioral range of 
instructional concern. The maximum possible effectiveness of instruc- 
tion is determined by the job validity of the terminal behavior pattern. 

The maximum possible efficiency of instructional control is determined 
by the degree of precision with which these two capabilities are specified. 

A tentative and rough outline of the steps involved in the construc- 
tion of an instructional program is presented in Figuie 1. In general, 
each step consists of an analytic process, a specification of the pro- 
ducts of the analysis, and a means for determining whether such products 
are adequate in terms of job-incurbent or student behavior. 



^apcr for Symposium, f 'Autor ated Teaching: Research Problems/’ 
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These stepi group themselves into three major phases. The first is 
joncemed with identifying the terminal behavior pattern that the 
stud nt actually needs to learn in order to perform the job adequately. 
The second phase is concerned with identifying transitio* ">1 and sup- 
porting associations , and with the organization of the to-be- learned 
behaviors into optimum practice units arranged in some optimum sequence. 
The third phase is concerned with the design and arrangement of instruc- 
tional situations, including the determination of response mode, type 
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of response, knowledge of results, and the design and integration of 
practice and review schedules. 

DETERMINING WHAT THE STUDENT NEEDS TO LEARN AND RETAIN 

The first phase in constructing an instructional program is directed 
toward identifying the behavioral range of instructional concern. This 
phase consists of two major steps: 

1. Determining the behavioral skills required for effective 
job performance. 

2. Determining the behaviors that the student needs to learn in 
order to produce the required job skills. 

The former is :pecified in the job description and the latter in 
the statement of instructional objectives. The job description lays 
out the behavior pattern required for effective accomplishment of the 
job in terms of the effects, accomplishments, or goals of the behavior 
with respect to the job environment . A major effect or goal can 
usually be analyzed into a set of sub-effects or sub-goals, which may 
in turn be analyzed into sub-sub-effects or sub-sub-goals, and so 
forth. Thus, we will generally find a hierarchical organization or 
breakdown Li descriptions of job skills. 

The source from which information is obtained for generating the job 
description will vary widely depending largely upon factors of availa- 
bility, time, and cost. Such information may be obtained by means of 
expert judgment or prediction, interviews and questionnaires , equipment 
analysis, gaming techniques, and so forth, and may or may not have been 
preceded by organization and position structuring activities. 

The first step in deriving the statement of instructional objectives 
consists of preparing a description or set of instructions for perform- 
ing the various job skills in sufficient detail to allow incoming 
students to perform the job activities, although not necessarily with 
the same speed and precision requirements placed on the job incumbent. 
This step essentially consists of a continuation of the breakdown or 
hierarchical analysis used in developing the job description. One 
major problem in this step, which also occurs in the development of 
the preceding job description, is that of selecting between alterna- 
tive procedures for accomplishing the same effect or skill. In select- 
ing one of several procedures as an instructional requirement, there 
are three general factors to be considered, job efficiency, ease of 
learning, and compatibility with related procedures. 

Each of these factors is in itself rather complex and each needs 
to be considered in relation to the others. The matter of compati- 
bility becomes especially important when we try to reduce to a minimum 
the number of new behaviors that the student will have to learn in 
order to be able to perform the job. I shall return to this factor. 

This first step, that is, developing a description of the various 
job skills that can be understood by the incoming student, provides us 
with a verbal linkage between the student's pre- instruct ion repertory 
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and the required job skills. Let us exsmine the nati 
linkage and how it relates to the instructional proc< 



udents will need 
are available to 



I am sure we will agree that in ino c t instances s* 
to learn considerably more than those behaviors that 
external observation on the job. Generally, thes' additional behaviors 
are implicit verbal and symbolic behaviors that we i'lfer occur on the 
job as a means of mediating the externally observabi:? job behaviors. 
Simply granting that such implicit verbal behaviors . re important to 



Simply granting that such impl^.v v. 

obtaining adequate job performance, tells us nothing about how to go 
about identifying them so that they can be described? In order to 
better understand what implicit verbal behaviors nee*! to be described, 
we first have to challenge the basic notion that such behaviors are 
necessary in order to obtain adequate job performance. 



How can we identify these implicit verbal behaviors? What are 
their essential characteristics? What role do they play in instruc- 
tion and in job performance? Ihe answers that we can obtain to these 
questions at this time are at best primitive. Let us begin by taking 
a close look at three hypothetical instructional situations. 

Situation 1-Non-verbal instruction. In this situation the instruc- 
tor demonstrates an activity to his students and the students then 
attempt to imitate the demonstration. Such demonstration-imitation 
trials continue successively until the students can perform the 
activity without error. Such a method is obviously slow and 
greatly restricts the number of students that one instructor can 
handle. To handle all the instruction for a complex job in this 
fashion would be far too costly and time-consuming. 

Situation 2— Unlearned verbal instructions . In this situation the 
instructor is allowed to communicate verbally with his students. 

He provides them with verbal instructions for performing each step 
of the activity. It would seem reasonable for him to begin with 
the demonstration of t lie activity, after which he would ve; bally 
lead the students through the activity step-by-step, giving the 
detailed verbal instructions for each step immediately before per- 
forming it. His objective, however, is to lead his students to 
be able to perform the activity without his continued verbal 
guidance. Thus, he would gradually withdraw this help from suc- 
cessive practice trials until the students could perform the 
activity correctly without any verbal guidance. In terms of 
instructional feasibility and economy, this situation constitutes 
an obvious improvement over the first. 

Situation 3-Learned verbal instructions . In this situation the 
instructor requires his students to learn the Verbal instructions 
for performing the activity or part of the activity, and then to 
practice it by generating their own verbal instruction. This would 
appear to be the most advantageous instructional situation for 
imparting the great majority of skills that adolescents and adults 
may be required to learn. Ihe movements and acts by which such 
skills are produced infrequently constitute behaviors for the 
adult human. In addition, there are usually some verbal instruc- 
tions available that tan be used to elicit the movements and acts; 
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that is, they are generally under verbal control. Thus, the student 
may already possess the ability to produce the movements and acts 
as a consequence of verbal stimulation, if he is told what to do 
in sufficient detail, he can do it. Consequently, the major 
instructional problem will be that of having the student learn 
how to generate his own verbal stimulation or verbal instruction 
for correctly performing the skill. 



DESIGNING VER8AL INSTRUCTIONS 

Once we have made the decision to employ learned verbal instructions 
in the production of skill performance, there are certain factors we 
should take into account in designing the verbal instructions them- 
selves. bet us consider a hypothetical example, schematized in Figure 2. 
Suppose we have an activity or skill consisting of a sequence of 
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25 steps. To train a student to learn and remember the verbal instruc- 
tions for performing these steps in sequence without introducing any 
verbal behaviors other than those involved in telling him how to per- 
form each one singly, will require considerable repetition. Such a 
procedure tends to restrict the cues for initiating a step to the 
termination of the preceding one and, at least during early learning, 
does not restrict errors between remote steps in the chain. For 
instance, the student might easily confuse the second step with the 
18th one. The problem here is that the student has too many alterna- 
tives to remember in order to make a decision as to what to do next 
at any particular insiant. 

We can improve this situation by applying some notions borrowed 
from George A. Miller, Eugene Galanter, and Karl H. Pribram (1, 2, 3). 
First, we can contrive a verbal supra-hierarchy constructed sc that 
each step contains some number of component steps less than seven. 

In this way we restrict the alternatives that a student nseds to 
remember at any one instant to a number within the average span of 
immediate memory. Second, we design the step labels on entries in 
the hierarchy to be ,T rememberable. 11 This implies that they should 
be short and bear some clear relation to the subsequent terms in 
the hierarchy. 

If we now turn the hierarchical structure in Figure 2 on its side 
and slightly rearrange the terms without changing the relations between 
them, we see that what we have been doing is actually quite similar to 
old, familiar procedures for outlining. 

One characteristic of a good textbook is that it explicitly lays 
out this kind of skeletal organization. In addition, we frequently 
find that the better students in a class will actually learn this 
skeletal organization and if it is not explicit in the text, better 
students will generally tend to make it explicit. This kind of 
organization of verbal material is not new— textbook writes, instruc- 
tors, and students have been using it for centuries. I am simply 
attempting to state some of the reasons for organizing materials in 
this manner and the implications of these reasons upon how we do it, 
as a means of improving the organization of verbal material in our 
instructional programs. However, there are still many unresolved 
problems concerning the construction of such verbal hierarchies. The 
hypothetical example given is restricted to a single activity and for 
such single activities the construction of verbal hierarchies seems 
to be fairly straightforward. There are still some remaining ques- 
tions concerning the degree to which we can build similar hierarchies 
for controlling the performance of several similar activities and 
thus reduce the amount of verbal material that the student will have 
to learn, and with the construction of such hierarchies for controlling 
the performance of activities that consist of complex branching 
chains containing many choice points— rather than simple linear chains 
as in the example. 
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Example 1 
Rigging the VTVM 



/ 



Legend 

Major functions 

I. Ohns measurement 
II. AC voltage measurement 
III. Positive DC voltage me as are ./lent 
IV. Negative DC voltage measurement 



A. 



Hake INITIAL SETTINGS 



1 . 



Turn ON meter and position FUNCTION swirch 

a. PLUG in meter {AC LINE cord) 

b. Position FUNCTION switch 

major function switch position 



I 

II 

III 

IV 



OHMS 

AC VOLTS 
+ DC VOLTS 
- DC VOLTS 



2 . 



3 . 



4 . 



c. Check to insure that red pilot LIGHT at top of meter turns ON 
Select RANGE 

a. Position RANGE switch 

I - posit lcn RANGE switch to an OHMS position which places the anticipated 
reading near the center of the scale (between 10 and 100 on the OHMS 
scale). If approximate value is not available, position RANGE switch 
to RX100Q position. 

II - position RANGE switch to highest AC setting (500 volts on this meter). 
If certain that reading will not be more than a gi^en lov'er range, 
then that lower range can be used. 

Ill 6 IV - position RANGE switch to highest DC volts RANGE (1CT0 volts 
DC only) 

b. WARM UP 

I, II » III 6 IV - allow 30 seconds for meter to warm up before proceeding. 
Needle will fluctuate while meter is warning up* 

CALIBRATE METER 
a. Select PROBES 

I - OHMS (green) 6 COMMON (black) 

II - AC (red) & COMMON ( black) 

HI & IV - DC (black with large insulated prcbe) ' COMMON (black) 



b. INFINITY adjust 

I - with the two probes not touching, adjust OHMS 
needle is positioned exactly on infinity (oo) 
c* ZERO adjust 

I, II* III* 6 IV - "short" probes (i.e., hold test 
while adjusting ZERO Ab_' knob until the rotter 
exactly at zero (0) of appropriate scale 
CHECK meter settings 



ADJ knob until meter 
on OHMS scale 

probes tightly together) 
needle is positioned 



B. Connect PROBES 



C. READ meter scale 



D. 



Determine if reading is in TOLERANCE 
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E . Turn OFF r-'ter 



SUMMARY AND CONCLUSIONS 

Let me summarize the functions served by a statement of instruc- 
tional objectives: 

First, such statements should specify the explicit behaviors 
required by the job that students need to learn in order to perform it 
adequately. The identification of these behaviors frequently involves 
a complex and tedious procedure, but not one that is beyond ready 
comprehension— at least for jobs that are already in existence. 

Second, such statements should specify the implicit verbal and 
symbolic behaviors contrived to mediate job performance, facilitate 
long-range retention, and, in many instances, make instruction prac- 
ticable. The conduct of this step is much more difficult. We need to 
develop rationales and procedures for contriving various kinds of 
implicit behaviors. In addition, such procedures must lead to a 
minimum number of associations to produce a maximum effect. Although 
this is the area covered in this paper, the rationales underlying the 
example given are at best primitive and are restricted to but one 
kind of situation for which verbal mediation might be used; that is, 
a moderately long linear chain. 

Third, statements of instructional objectives should specify 
the relevant characteristics of the instruments to be used in evaluat- 
ing student achievement (4, 5). I have already mentioned the specifi- 
cation of the behaviors and behavioral associations that the student 
needs to learn and remember. In addition, it is necessary to specify 
the objects and the proper situations for each act or operation and 
the criteria of adequacy for each act or operation. Procedures for 
sampling content and for scoring should also be specified. What is 
desired here is a procedure for specifying how a particular test 
should be built so that separate test constructors working independ- 
ently can each construct a test that will lead to the same decision 
concerning the achievement of individual students. 

Traditional test construction rationales and procedures arc rot 
appropriate to a measurement of in-course achievement. Traditional 
procedures have been developed primarily for selection situations in 
which the primary problem is one of ordering individuals along some 
achievement dimension. In an instructional situation, however, our 
primary interest is in determining whether or not a given individual 
has attained a specified achievement criterion. The fact that some 
"passing" students learn more than other "passing" students is of 
little concern to us unless we are going to base some action on this 
difference. There is little point in differentiating between students 
unless they are to be treated differently. 

What does all this have to do with automated teaching and pro- 
graming? If you will look again at Figure 1, I think the answer will 
become apparent. Both the reinforcement and the communication models 
currently underlying the programing movement are primarily concerned 
with the mechanics of learning. Neither is particularly concerned 
with the design of the terminal behavior pattern or with the design 
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Example 2 



7r.,> following items are to answered in order with no retracing. Aul items must be 
anuweied correctly. 



1. List in any order the four major functions for which a VTVM may be used. 

a . ohms measurement 

b. AC voltage measurement 

c. positive DC voltage measurement 

d. negative DC voltagement measurement 

2. List in order the five common steps which need to be performed in order to use the 

VTVM for an y of its major functions. 

a. make initial settings 

b. connect probes 

c. read meter scale 

d. determine if reading is in tolerance 

e. turn off meter 

3. List in order the four common steps which need to be performed in order to make 

the initial settings for any of the major functions of the VVVM. 

a. turn on meter and position function switch 

b. select range 

c. calibrate mater 

d. check meter settings 

4. List in order the three common steps which need to be performed in order to turn 

on the meter and position the FUNCTION switch . 

plug in the meter AC line cord 

b. position function switch 

c. check to insure that red pilot light at top of meter turns on 

5. Match the following FUNCTION switch positions with the appropriate function. 

a . chms measurement 

b- AC voltage measurement 

c. positive DC voltage measurement 

d. negative DC voltage measurement 

a. +DC VOLTS 

b. -DC VOLTS 

c. OHMS 

d. AC VOLTS 

c a ■ 

_d_ b. 
a c. 

_b_ d. 



of transitional and supporting associations. If these aspects of the 
process arc not explicitly controlled, then they are passed by default 
to tiic whims of individual programers. Furthermore, the failure to 
develop specific rationales and procedures for these aspects of the 
process may well be or.c reason why programing is an expensive, time- 
consumi ig art form. The programer is left with the overwhelming prob- 
lem of doing all of these steps simultaneously. Although every how- 
to-do-it article on programing that I have seen starts with an instruc- 
tion to "define the objectives," I have yet to see one that tells exactly 
how to go about doing this. 
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MILITARY CONTROL— A FREQUENTLY MISSED TRAINING OPPORTUNITY 1 



Robert G. Smith, Jr. 



Advantages of military control in motivating students 
to be rapid learners in self-paced programmed instruc- 
tional courses are discussed. Examples of techniques 
to control student learning behavior are given. 



I am concerned by the theme of this symposium, and particularly by 
the preliminary material sent to participants. This material says in 
part: "The participants on this symposium will discuss how to cope 

with administrative problems sure to be generated by allowing trainees 
in closely scheduled training systems (most obviously, the military) 
to proceed at their own paces. What suggestions should be made to 
practical-minded commanders who find a lot of bright people on their 
hands, having completed an eight-week course in two weeks?" 

The greatest disservice we can do to "practical-minded commanders" 
is to cut the heart out of one of the greatest advantages of program- 
med instruction. This advantage is the opportunity for the bright 
people to complete an eight-week course in two weeks. Insteaa of con- 
sidering this a problem, let us consider it an opportunity— an oppor- 
tunity for mobilizing military controls over students so as to motivate 
the best students to make the most rapid progress of which they are 
capable through the training program. 

I would like to tell a story told to me several years ago by a 
pilot instructor during World War II. From his familiarity vith flying 
students my instructor friend knew that the big event of the training 
program was the cross-country flight in which a considerable amount of 
freedom was given to the student to select his destination. My friend 
convinced his superiors to try a rather daring deviation from the usual 
training practice. He promised the students that if they reached the 
stage at which they were ready to go on their cross-country flight 
before a certain time (which was less than the usual time it took the 
students to be ready), then any time saved they could have as leave to 
take at the place to which they flew. He gave his students instruction 
in the use of mental practice and "turned them loose." 



] Paper for American ^Psychological Association Convention, New 
Yc ~k City.. September 191*, as part of a symposium, "Problems in 
Classroom and Institutional Management Created by Teaching Machines 
and Programmed Instructional Materials." 
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The result of this procedure was that the students were able to 
make their cross-country flights in a great deal less time than was 
the case in the regular instruction. 



The story illustrates my theme very well. The military services 
typically exert a high degree of pervasive control over the behavior 
of their members. At the same time these control techniques are 
seldom mobilized on the side of training. 

With conventional lock-step training procedures, there is often 
little gain for the better student if he progresses and learns more 
readily than the poorer one. But one of the principal characteristics 
of automated instruction is that it provides for the possibility of 
the better student’s completing a program much sooner than the poorer 
student. Rather than discuss the problems created by this character- 
istic, I would prefer to discuss some of the opportunities that are 
now available to the military services if they will appropriately 
mobilize their controls to assist training programs. 

All too frequently the various privileges that may be available 
to military personnel are not dispensed by the persons in charge of 
the training programs. Instead they are dispensed by someone else — 
a student detachment commander, whose functions are typically those 
of an administrator and housekeeper. Let’s examine some of the 
possibilities that exist for mobilizing military controls in order 
to motivate students to progress through an individualized training 
program as rapidly as possible. 

One of the standard military ways of rewarding good performance 
is the granting of a pass. In lock-step training programs students 
typically are not granted passes because of the problem created when 
students miss classes and need to ,p make them up." However, in an 
individualized training program a pass could be a worthwhile reward 
for a student who is making progress at a very rapid rate. 

Another possibility exists in the matter of assignments . Normally, 
when a group of students is entered into a training program, there 
is also a group of assignments awaiting these students upon the com- 
pletion of their training program. It appears to me that an excel- 
lent motivator to get people to complete a training program quickly 
would be to provide that personnel who complete the program will be 
given their choice of those assignments available at the time they 
finish. Thus, the person who finishes first can have his choice 
from among all the assignments that are available to his class. 

Those who finish later will find that the choicer assignments will 
have been taken by those people who got through earlier. 



Another characteristic of the life of the typical student is that 
the military services tend ^o keep him occupied at some kind of 
activity all day long and often well into the night. As a result, 
the student seldom hus free time to catch up on his sleep, or to take 
care of other kinds of personal needs. If he is making exceptionally 
rapid progress in the training program, he might be permitted to report 
to the classroom or to the place of instruction later, or to leave an 
hour or so earlier, than the rest of the students. This would give 
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rapid learners an opportunity to get in a little extra sleep, to take 
care of personal business, or just to relax a little. 

Another frequent characteristic of military training programs is 
the compulsory study hall. Students who are making exceptionally rapid 
progress through the course nay well be excused from such study halls* 

I think I have given enough examples of specific possibilities. I 
r eel sure that ingenious training officials can think of a large number 
of ways that military controls could be used to support a training pro- 
gram, M/ main point, however, is that with lock-step training programs 
we could not effectively make use of these techniques for motivating 
students. But with training programs in which the student progresses 
at his own rate, the military services, through their control of stu- 
dent behavior, have a very significant opportunity to mobilize these 
controls in order to greatly assist their training programs. 



A RATIONAL ANALYSIS UF 
THE PROCESS OF INSTRUCTION 1 

P.G. Whitmore 



Summary — Instruction is defined as a process for controlling 
student behavior so as to insure student learning, rather than 
as a process for merely presenting information to students to 
learn in whatever way they can. Learning, in turn, is defined 
in terms of behavioral associationism . Thus, the problem of 
specifying an instructional program so as to lead to effective 
control of student learning as directly as possible is largely 
a problem of adequately describing the behaviors required of 
the student at specified points in the instructional program. 

The main body of the paper is concerned with the general appli- 
cation of these basic definitions to the problems of: 

(1) Identifying what it is that is to be learned, 

(2) Sequencing the order in which instructional materials 
are to be presented, 

(3) Designing instructional situations for accomplishing 
the desired learning. 

Evaluation is defined as consisting of two major aspects: 

(1) Evaluation of the effectiveness of the instruction for 
inculcating students with the behaviors selected for 
them to learn, 

(2) Evaluation of the effectiveness of the behaviors selected 
for the student to learn for producing adequate job 
performance . 



INTRODUCTION 



The purpose of this paper is to provide the lay reader with a 
general understanding of the process of instruction, and an apprecia- 
tion of the problem areas that must be resolved before we can claim 
to have a complete and adequate technology of instruction. The body 
of the paper is organized as follows: 

I. What is instruction? This section presents a borrowed 
definition of learning and relates instruction to learn- 
ing. Furthermore, it attempts to answer the question, 
How should instructional programs be specified? and 
draws a distinctiun between methods of instruction and 
methods of presenting information. 



^is article appeared in IRE Transactions on Education , 
1961, pp. 135-143. 
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II. The major steps in the construction of mass instructional 
programs . This section is concerned with the problems of: 

A. Identifying the material to be learned. 

B. Sequencing the order in which instructional materials 
are to be presented. 

C. Designing instructional situations for accomplishing 
the desired learning. This subsection is broken down 
into more detailed considerations, dealing with 
obtaining initial comprehension of instructional 
materials and long-range retention, and minimizing 
learning interference. 

III. Evaluation . This section deals with evaluating both stu- 
dent learning and the program of instruction. 

The author T s primary concern is with constructing and conducting 
mass instructional programs for adolescents and adults. In this kind 
of program: (1) The major problem is that of assembling skills from 

previously learned behavioral components. (2) The control of learning 
is paramount over the presentation of information. (3) The cost of 
controlling learning can be justified in terms of expected savings 
from improved instruction. (4) The construction of the program will 
probably involve coordinating the activities of several different 
groups of workers. Although many of the principles, techniques, and 
strategies appropriate to this kind of instructional situation are also 
appropriate to others, it is well beyond the scope of this paper to try 
to spell out all the differences and similarities that may be involved. 
Consequently, any generalizations from this situation to related situa- 
tions are made purely at the reader’s discretion. 

I. WHAT IS INSTRUCTION? 

How Is Instruction Related to Learning ? 

Instruction is effective only to the extent that the desired learn- 
ing occurs. If none of the desired learning occurs, then we are forced 
to conclude that no instruction has occurred. The late Edwin R. 

Guthrie 1 dr" ,ied learning as ’'changes in behavior which follow behavior. 1 ' 
Thus, learning consists of responding ’'differently to a situation because 
of past responses to the situation.” Consequently , during learning the 
student will be engaged in the production of behavioral responses. 
Instruction, on the other hand, is concerned with the arrangement of 
situations designed to elicit from the student those behaviors that it 
is desired that he learn. The production of behavioral responses by 
the student is the integral constituent of both learning and instruction. 
Or as Guthrie 2 noted, ”A student does not learn what was in a lecture or 
book. He learns what the lecture or book causes him to do.” 



^F.R. Guthrie, " The Psychology of Learning,” (Revised Edition) 
Harper and Brothers, New York, N.Y., 1952. 

2 , ”Condit ioning: A Theory of Learning in Terms of Stimulus, 

Response, and Association.” The Psychology of Learning , Natl. Soc. 
Stud. Educ., 41st Yearbook, part 2, 1942, pp. 17-60. 



Guthrie's definition of learning does not imply purpose or improve- 
ment, but simply "changes in behavior which follow behavior." Purpose 
is imposed upon learning as a consequence of the behaviors that are 
selected for the student to learn. Improvement is imposed upon learning 
as a consequence of the arrangement of situations designed to evoke the 
behaviors selected for the student to learn. Thus, both purpose and 
improvement are parts of the process of instruction which are not inher- 
ent in learning- Instruction is specifically concerned with providing 
purpose to learning, and improvement in learning, with respect to the 
designated purpose* 

Behavioral Descriptions and the Specifications 
of Instr uctional Programs 

The problem of specifying an instructional program that will lead 
as directly as possible to effective control of student learning is 
largely a problem of adequately describing the behaviors required of 
students at specified points in the instructional program. Behavior, 
however, is a difficult thing to describe, not only because it is con- 
tinuously changing, but also because there are so many different char- 
acteristics of it that we could select to describe at different levels. 
For instance, we might describe muscular and anatomical movements and 
glandular responses in such minute detail that a description could be 
generated only after the fact, and could fit only the single behavioral 
events for which it was generated; that is, the behavior would be non- 
repetitive in that we could never expect to observe another behavioral 
event that would fit every detail of the description. Or we might 
instead describe behavioral acts in terms of gross segments along var- 
ious descriptive dimensions applicable to muscular and anatomical 
movements and glandular responses. Since such descriptions could fit 
many behavioral events, we could use them to denote repetitive behav- 
iors, and we would thus be provided with at least the possibility of 
predicting behavior in future situations. This, of course, is the 
basic goal of behavioral science. Note, however, that there is no 
suigle level at which behavioral acts might be described. A greater 
number of behavioral events will fit gross descriptions than will fit 
detailed descriptions; consequently, gross descriptions will be more 
repetitive and easier to predict. And they will generally possess 
less practical utility. 

Instead of describing behavior in terms of characteristics of 
muscular and anatomical movements or acts, we might find it more use- 
ful in some instances to describe it in terms of its effects, accom- 
plishments, or goals. This kind of description we designate as 
behavioral skill rather than as behavioral movements and acts. The 
goa 1 constitutes the intrinsic characteristic of descriptions of skill. 
Skills may also be described at many different levels of generality. 

And, again, the grosser the level of description, the more repetitive 
and easier to predict will be the skill and the less its practi- 
cal utility. 

There is another aspect to describing behavior that we must also 
consider. This is the problem of who does the describing. Suppose 
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that we were observing Joe B., an engineer, sitting in his swivel chair 
with his feet, propped up on his desk. We might observe that Joe scratches 
his right ear with his right hand, runs his right index finger around the 
right side of his neck under his collar, convulsively jerks his left foot 
three times, and shifts his weight on his buttocks, in sequence. Our 
report of Joe’s behavior would probably not significantly impress his 
supervisor. If, however, we were to ask Joe to describe what he had been 
doing during this same period of time, he might reply, "Well, I was mull- 
ing over this circuit design problem we have been having on equipment X 
and I think I f ve got it licked." He might then proceed to put some dia- 
grams, symbols, and calculations on the blackboard to substantiate his 
reply. This description of Joe’s behavior would probably make a signifi- 
cant impression upon his supervisor. That which is described m?y be 
quite different, depending on whether it is described by an external 
observer or by the behaving organism itself. Ihere are some aspects of 
behavior which are not generally open to external observation. It is, of 
course, debatable whether these unobservables (t.6., thinking) constitute 
behavior. I contend that they do. Furthermore, there are instances in 
which some of these unobservables become observable. Though most of us 
go through long years of being trained not to externalize our thinking, 

I must confess that I do occasionally get caught at it, For instance, 
frequently when I am alone I do my "thinking" out loud, or at least 
engage in some small but perceptible throat, tongue, and lip movements. 

It should be obvious by now that it is not entugh simply to exhort 
educators and training specialists to describe jobs and instructional 
programs in terms of behavior, for still unanswered are such questions 
av whether to describe behavior in terms of movements and acts cr in 
terms of skills, in terms of the behaving organism or in terms of an 
external observer, and at a gross or a detailed level, All these modes 
of description are appropriate, but for different purposes. In general, 
the behaviors that are selected for description (f.e., those that stu- 
dents in a program of instruction will be required to learn) are selected 
with respect to the behaving organism. For instance, in building an 
instructional program for the training of engineers, we would select 
those behaviors which are of significance to practicing engineers whether 
or not they are available to external observation. The student, however, 
may be required on occasion to externalize his "thinking" so that the 
instructor can either evaluate his progress or decide what instructional 
presentation to make next. Thus, for purposes of controlling instruc- 
tion, behavior should be described with respect to an external observer. 
Whether the student "thinks" to himself or talks out loud may ha^e litvle 
effect upon his actual learning, although talking out loud will generally 
be more time consuming than "thinking." The instructor, however, cannot 
evaluate the student’s progress or alter his instructional presentations 
unless the student engages in relevant externally observable behavior. 

Descriptions of behavior in terms of skill are necessary so that the 
instruction can impose purpose or direction upon student learning and 
provide for improvement towards that purpose. Without the specifica- 
tion of skill requirements, we would have no way of determining the 
adequacy of the movements and acts actually practiced and learned by 
the student. The student, however, does not directly learn the effects 



of movements and acts; instead, he learns the movements and acts that 
produce the effects. Consequently, descriptions of the movements and 
acts are generally needed as a basis for designing the instructional 
material actually used by the student. 

The most detailed level at which each of these behavioral charac- 
teristics should be described depends largely upon the ability of the 
users of the descriptions to comprehend them correctly. For instance, 
the ultimate users of movement-and-act characteristics of behaviors 
are the students themselves, and the level of detail at which these 
characterist ics should be specified depends primarily upon the level 
of comprehension of the incoming students, that is, previous education, 
training and experience, abilities and aptitudes, and so forth. In 
fact, it may not always be necessary to specify the movement -and-ac 1 ; 
characteristics of all the behaviors required for a given job. Students 
may enter the instructional program with the movements and acts neces- 
sary for producing some skill effects already in their repertory. This, 
for instance, is true of the verbal behaviors required in the great 
majority of instructional programs for adolescents and adults. Thus, 
the major instructional problem may be, and generally is, one of assem- 
bling already possessed skills into more complex skills. 

The preparation of behavioral descriptions for instruction purposes 
Will frequently involve several groups of workers, that is, task analysts 
to prepare skill descriptions of the job with regard to an external 
observer; training analysts to derive and prepare movement-and-act and 
skill descriptions of the job with regard to the behaving organism, on 
the basis of the task analysts 1 descriptions; and instructors and techni- 
cal writers to prepare instructional material on tho basis of training 
analysts' descriptions. In each instance, that which is described and 
the level at which it is described will depend upon the needs of the next 
group in the sequence to perform its part of the design of instruction. 

Thus we see that in deriving and specifying the behaviors requisite 
for a given instruction program, there are several characteristics of 
behavior with which we should be concerned as a basis for determining 
what needs to be learned and for effecting adequate control of such 
learning as follows: 

(1) Movement and act descriptions. 

(2) Skill descriptions. 

(3) Descriptions with respect to an external observer, 

(4) Descriptions with respect to the behaving organism. 

(5) Different levels of descriptive detail. 

Dist i nguishin g Between Pre sentation a nd Instruction 

Most traditional methods of mass instruction, lectures, conven- 
tional textbooks, demonstrat ions, are for the most part simply methods 
for presenting information to students. Although the presentation 
of information is an important aspect of instruction, it docs not 
in itself constitute a complete method of instruction. 1. earning 
must inevitably involve the production of some kind of behavioral 
response by the student. Effective instruction not only presents 
information to students, but also attempts to present information 
in such a manner as to lead the students into making the desired 
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behavioral responses. Dressel^ has bluntly stated, "Classroom practices 
which are restricted to textural or teacher presentation of knowledge 
and the testing of the extent of recall of this knowledge are unworthy 
of the name of instruction. . . . The teacher covers content but does 
not instruct students. The majority of students remain completely pas- 
sive, and work only to memorize what the teacher emphasizes . " 

This distinction between instruction and the presentation of infor- 
mation is one which should be belabored, in view of the widespread 
belief that education’s great leap into the future can be made via mass 
information media such as films and television. Such techniques pri- 
marily expand the lecture or demonstration beyond the walls of the 
classroom, but do not in themselves constitute complete instructional 
methods; that is, they do not attempt to control student behavioral 
responses in such a way as to maximize learning effectiveness 
and efficiency. 

II- THE TAJOR STEPS IN THE CONSTRUCTION OF 
MASS INSTRUCTIONAL PROGRAMS 

As previously indicated in the Introduction the major steps in 
the construction of mass instruction programs are: 

A. Identifying the material to be learned. 

B. Sequencing the order in which instructional materials are 
to be presented. 

C. Designing instructional situations for accomplishing the 
desired learning. 

A. I dentifying the Material To Be Learned 

A complete instructional program that does in fact effectively 
control the student’s learning may not necessarily be adequate if it 
does not inculcate behaviors in the students which are pertinent to 
the job situations in which they will eventually be required to per- 
form* For instance, an electronics maintenance program that teacher 
its sludents how to design electronic circuits rather than how to 
troubleshoot them will turn out inadequate job holders regardless of 
the effectiveness of its instructional methods. Improving the effec- 
tiveness of instruction in such a program would only lead to the more 
efficient production of equally incompetent graduates. 

In constructing instructional programs for producing graduates 
who are capable of performing a given job, it is patently obvious 
that in most instances the students will need to learn considerably 
more than just those behaviors that are available to external obser- 
vation on the job. Certainly, an electronics technician needs to 
know something about "theory 11 and "troubleshooting strategies." Such 
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verbal behaviors are not available to external observation during 
normal performance of the job. Simply granting that such unobservable 
verbal behaviors are important to obtaining adequate job performance, 
or that job behaviors should be described with regard to the behaving 
organism rather than with regard to an external observer, tells us 
nothing about how to go about identifying these behaviors so that 
they can be described. In order to understand better what unobserva- 
ble verbal behaviors need to be described, we must first challenge 
the basic notion that such behaviors are necessary for obtaining ade- 
quate job performance. 



Identifying One Role of Unobservab le Ve rbal Behavior s 

How do we identify these externally unobservable verbal behaviors? 
What are their essential characteristics? The answers that we can 
obtain to these questions at this time are at best primitive. Let us 
begin a search for answers to these questions by taking a close look 
at three hypothetical instructional situations. 

Situation 1 . Nonverbal instruction: The instructor demonstrates 

a behavioral activity to his students, and the students then attempt 
to imitate the instructor’s demonstration. Such demonstration- imitation 
trials continue until the students can perform the activity without 
error. Such a method is obviously slow and greatly restricts the num- 
ber of students that one instructor can handle. To handle all the 
instruction for a complex job in this fashion might well consume the 
entire lives of both the students and the instructor— -and even that 
length of time might not be sufficient. 

Situation 2. Unlearned verbal instructions : The instructor is 

allowed to communicate verbally with his students. He provides his 
students with verbal instruction for performing each step of the 
activity. It would seem reasonable for him to begin with a demonstra- 
tion of the activity, after which he would verbally lead his students 
through the activity step by step, providing them with detailed verbal 
instructions for each step immediately before its performance. His 
objective, however, is to enable his students to perform the activity 
without his continued verbal guidance. Thus, he would gradually with- 
draw the verbal guidance from successive practice trials until the 
students could perform the activity correctly without any verbal 
guidance whatsoever. In terms of instructional feasibility and econ- 
omy, this situation constitutes an obvious improvement over the first. 

Situatioyi 3* Learned verbal ins true tioy'is: The instructor requires 

his students to learn the verbal instructions for performing the activ- 
ity or part of the activity, and then to practice it by generating 
their own verbal instruction. This would appear to be the most advan- 
tageous instructional situation for imparting the great majority of 
skills that adolescents and adults may be required to learn. The move- 
ments and acts by which such skills are produced only infrequently 
constitute new behaviors for the adult human. In addition, there are 
generally some verbal instructions available which can be used to 
elicit the movements and acts; that is, the movements and acts are 
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generally under verbal control. Thus, the student may already possess 
the ability to produce the movements and acts as a consequence of 
verbal stimulation; that is, if he is told what to do in sufficient 
detail, he can do it. Consequently, the major instructional problem 
will be that of having the student learn how to generate his own verbal 
stimulation or verbal instructions for correctly oerforming the skill. 

Soma Factors Affecting the Learning o f 
VerbaT Instruction s 

Once we have made the decision to enploy learned verbal instructions 
in the production of skilled performance, there are certain factors 
which we should take into account in designing the verbal instructions 
themselves. Let us consider a hypothetical example. Suppose we have 
an activity or skill consisting of a sequence of 25 steps. To train a 
student to perform (that is, both learn and remember) these 25 steps 
in sequence without introducing any verbal instruction other than that 
involved in telling him how to perform each step singly would be time 
consuming and costly. Such & procedure would tend to restrict the 
cues for initiating a step to the termination of the preceding step 
and, at least during early learning, would not restrict errors between 
remote steps of the chain. For instance, the student might easily con- 
fuse step 2 with step 18. How might the verbal controls be better 
designed to facilitate learning and retention? 

Let us seek an answer to this question by first drawing an analogy 
between human beings and computers— an analogy quite similar to one 
made several years ago by Miller. 1 The construction of a system of 
verbal controls for facilitating learning and retention is in many 
ways akin to the construction of a retrieval program for a computer. 
Although the human computer has tremendous capacity in its permanent 
storage facility, it does not possess random access. Access to this 
permanent storage generally has to be nade through a working memory 
of very limited capacity. This limited capacity wonld appear to be 
determined by the span of immediate memory. The average span of 
immediate memory over a wide variety of materials and people is 
around 7 items. 2 Thus, we want to design verbal contiol in such a 
way that there will never be a need for more than 7 items in the 
working memory in any one instance. Since 7 is an average figure 
over people and materials, we would in practice generally want to 
restrict the maximum number of items ever actually needed in the 
working memory at any one instant to some number less than 7, so as 
not to operate the working memory continuously at maximum capacity, 
or beyond the maximum of many students. 




^.A. Miller, "Human Memory and the Storage of Information, M IRE 
Trans, on Information Theory > vol. IT-2, September 1956, pp. 129-137. 

2 G.A. Miller, "The Magical Number Seven, Plus or Minus Two: Some 

Limits on Our Capacity fer Processing Information," Psychol. Rev. s 
vol. 63, 1956, pp. 81-97. 
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The Organization of Verbal Instructions 



We can and frequently do reduce the load on the working memory by 
organizing verbal material into a hierarchical arrangement or outline. 
For example, the tables of contents of many books (particularly many 
published before the twentieth century) are organized into an outline 
format with heads and subheads, topics and subtopics, and so forth. 

In order to see what role this restriction on the span of immediate 
memory plays in our development of verbal controls for skilled perform- 
ance, let us again consider the hypothetical activity consisting of 
2S discrete steps, as diagramed in Figure 1. The bold-faced numbers 
from 1 to 25 at the base of the triangle represent the externally 
observable motor nonverbal) acts required to accomplish each 

step in the activity. The matching light-f3ced numbers immediately 
above the bottom row represent the specific verbal instructions neces- 
sary for eliciting the motor acts required to accomplish each step in 
the activity. 

I have already indicated that an instruction method which would 
simply require students to learn the motor acts in the individual 
steps as a sequence with no verbal controls would not be a particularly 
efficient method of utilizing both time and money. The introduction of 
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verbal instructions for each step doss constitute an improvement, but 
it still would not be particularly efficient simply to require that 
the student learn the verbal instructions as a simple chain of verbal 
behaviors. However, we can break this sequence of 25 steps in sub- 
groups in which each group would consist of 7 or fewer steps. Thus, 
in selecting the steps which he is to perform next the student need 
not consider more than 7 items at any one time. Such grouping, of 
course, should not be completely arbitrary. Steps that seem to have 
some common function or goal should be grouped together. Thus, in a 
real situation, we would probably not have the same number of steps in 
each group. The common function of each group of steps becomes a label 
for that group and this label in turn constitutes a step in a sequence 
described at a grosser level. 

Now suppose that the entire sequence has a general function or 
overall goal, A. Then the names or labels given to cur groups are in 
tact subfunctions or suboperations v/hich must be accomplished in order 
to obtain the general function A . Our system of verbal controls thus 
assies a hierarchical arrangement. This is the satoe typ r . of arrange- 
ment that aas been proposed oy Miller, Galanter, and Pribram 1 for the 
human storage of what they term plane . 

If we now turn the hierarchical structure in Figure 1 on its side 
and rearrange the terms slightly, without changing the relations among 
them, we sea that what we have been doing is actually quite similar to 
old familiar procedures for outlining. The nain thing we have done so 
far is to place a restriction upon the number of elements in the cate- 
gories of our outline. It becomes obvious that there is also a second 
restriction to be placed upon outlining procedures. The entries them- 
selves must be designed so as to be 1P rememberable . " This implies that 
they should be short and should bear some clear relationship to the 
succeeding terms in the outline or to the motor acts which they are 
to elicit. 

Some readers may be of the opinion that such outlines are trivial 
and are certainly not a significant aspect of the learning requirements 
laid on the student. Let me answer this charge by asking these same 
readers to reflect back on their own experiences. When faced with a 
particularly difficult or complex section of textual material, how many 
of you have deliberately developed an outline of the material specifi- 
cally to facilitate your learning and remembering of it? Certainly 
the outline does not constitute all that a student may be required to 
learn, but it does provide him with a basic framework of associations 
within which he can organize and integrate other associations. 

The organization of verbal material that I am suggesting is not 
new. Tex.took writers, instructors, and students have been doing this 
ki'.id of thing for centuries. I have simply attempted to present some 
of the reasons for organizing materials in this manner, together with 
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the implications of these reasons for improving the organization of 
verbal material in instructional programs, However , there are still 
many unresolved problems concerning the construction of such verbal 
outlines or hierarchies. The hypothet ica] example that I have given 
is restricted to a single activity, and for such single activities, 
the construction of verbal hierarchies appears to be pretty straight- 
forward. Still unanswered are questions concerning the degree to which 
we can build similar hierarchies for controlling the performance of 
several similar activities, and thus reduce the amount of verbal mate- 
rial that the student will have to learn; and concerning the construc- 
tion of such hierarchies for controlling the performance of activities 
that consist of complex branching chains containing many choice points, 
rather than simple linear chains as in the example. 

Th ? Advantages of implo yinu Verbal Controls 
of Job Performa nce — 

There are many advantages which accrue iO the use of well-designed 
verbal behaviors as a means of controlling skill performance, If the 
discrete motor acts required for skill performance already exist singly 
in the behavioral repertory of incoming students, so that the major 
learning problem is that of assembling these motor acts into some proper 
sequence for producing the skill, then assembly and sequencing can 
probably be done most efficiently and effectively by means of verbal 
controls (t.e., learned verbal instructions for performing the si ill). 
Verbal behaviors can generally be practiced more rapidly than c?v motor 
behaviors. Thus: (1) The verbal behaviors would allow us to have More 

practice per unit time than could otherwise be obtained. (2) We could 
reduce the amount of time which individual students would have to spend 
practicing on expensive training devices whose use requires the produc- 
tion of motor acts. The sequencing of motor acts could be learned 
verbally away from such devices, (3) The initial comprehension and 
leng-range retention of verbal material c?n be greatly facilitated by 
adroit use of associative supports, such as inalogies and mnemonic 
(memory) devices. (4) It is possible to build generality into verbal 
systems so as to provide for greatly increased transfer of training 
from one job situation to another. 



B . Sequencing the OrJ e r in Which Behaviors 
Are To Be Learned 

The order in which behaviors are learned greatly affects the effi- 
ciency and effectiveness with which such learning is accorrpl ished , few, 
if any, of us would disagree with this statement. But it says nothing 
about what order is most efficient and effective, nor does it indicate 
what characteristics of behavior should be considered in determining an 
efficient and effective order. 
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A general solution to these problems begins to form when we carefully 
examine the behaviors that we wont our students to learn. We are not 
interested in having the specified movements and acts occur on the job 
in a random manner. Rather, we want each to occur only in circumscribed 
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stimulus situations; that is, we want to establish specific associations 
between stimulus conditions and movement-and- act responses. An associa- 
tion consists of a stimulus situation or cue and a behavioral response 
occasioned by that cue. The response may in turn produce sensations 
which can be used as a cue for occasioning another response. Thus, we 
can have associative chains of responses in which the first response 
produces the cue for the second, the second produces the cue for the 
third, and so forth. One important fact about associations and asso- 
ciative chains that we should keep in mind in that they possess direc- 
tion; that is, a given cue must precede a given response, which in 
turn produces the cue for a subsequent response, and so forth. 

In the verbal hierarchies or outlines mentioned above (Section II-A), 
there is not a simple chain of responses but rather a net of responses. 

In order to transform this net of responses into a net of associations, 
we need to determine the source of the cue for eliciting each response 
in the net. 

In actually using a learned verbal hierarchy on the job, the stu- 
dent will enter it at the top and work down to the bottom, rather than 
entering at the bottom and working to the top. In addition to using 
the associations from top to bottom, he will also need to use the 
temporal associations from left co right. Thus, the cue for any given 
response will always appear in the hierarchy to the left of or above 
the response. Consequently, for any given hierarchy, student practice 
will proceed from whole to part, or from general to specific, and 
will maintain the temporal associations in order at each level of 
the hierarchy. 

The learning requirements for a job will frequently consist of at 
least several associative hierarchies that are relatively independent 
of each other; that is, they might be viewed as subjobs within the 
overall job. Generally, we will want to sequence these subjobs so as 
to minimize the amount of new learning required of the student in 
going from one to the next. Thus, the first subjob that the incoming 
student would be required to learn would be the one that he could 
learn the most easily and quickly, the second subjob would be the one 
that the student could learn most easily and quickly having already 
learned the first, and so on. 

C. Designing Instruct ional Sit uations for Accomplishing 
and Insuring the Learnlng~of Specified Behaviors 

As indicated in the Introduction, this subsection will deal with 
three problem areas, as follows: 

Obtaining initial comprehension. 

Obtaining long-range retention. 

Minimizing interference with learning. 

Ob taining Initial Comprehension 

Since the entire program cannot be learned at once, the first 
problem is to select the size of the learning or practice unit. On 



the one hand, we want to avoid units that are so large that the student 
is unable to retain the material presented in the early part of the 
unit until the entire unit has been presented. On the other hand, we 
want to avoid units that are so small that on;e the student has learned 
them he still faces a major learning requirement in having to learn to 
put them together. This is the old problem o:: whole-versus-part prac- 
tice and, unfortunately, there is no simple solution for it. There are 
many factors that enter into a decision concerning the size of the 
practice unit; of these, the following are probably most important; 

(1) The nature of the material. 

(2) The general intelligence of the .jitudent, 

(3) Ihe student's previous education and experience. 

(4) The particular stage of practice at which the student 
has arrived. 

In very general terms, largev units would appear to be more effec- 
tive with brighter, better-educated students, with material having a 
definite sequence or organization, and in the later stages of practice. 
Several different methods for solving this problem have been suggested, 
of which the most generally applicable is a "whole" method which allows 
for extra lepetition of difficult pa* ts withii the context of the 
whole. The basic problem, of course, is not whole versus part, but 
rather "How big a part?" Other factors also seem to play a role in 
determining the size of the practice unit, but they are factors that 
are extremely difficult to identify or define, For instance, a factor 
grossly connoted associative cohesiveness may have a bearing on 
this problem. 

It will frequently not be possible, using a single instructional 
presentation, to lead all of the students inio producing all the 
behaviors involved in a single practice unit. A series of instructional 
presentations preceding the first presentation of the complete practice 
unit may be necessary if all of the component behaviors do not already 
exist in the students' behavioral repertory. Instruction is usually 
concerned with assembling the behavioral act; out of component move- 
ments or out of acts already in the students 1 repertory. This in 
itself may be a many-step process involving the assembly of small acts 
into la; ^er acts, which in turn are assembled into still larger acts, 
which in turn are assembled into the complex skills called for by the 
job. The problem is further complicated by the fact that all students 
do not possess precisely the same initial repertory. Thus, some stu- 
dents may have to be started at a very detailed level and others at a 
more gross level. To complicate the picture yet further, some behav- 
iors defined as skills rather than as movements and acts may be ade- 
quately accomplished by any of a wide variety of movement-and-act 
combinations. Because of wide individual differences in the students' 
initial behavioral repertories, some of thes* combinations might be 
more readily learned by some students, and ohers by other students, 
Thus, it may be necessary to lead different ..ludents up to each com- 
plete practice unit by different routes. 
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Obta ini eg Lcmg-Ra n ge Retention 

Our concern is not simply to obtain adequate performance from stu- 
dents during and at the end of instruction, but also to obtain adec ’ate 
perf orra&nce in an ultimate job situation after formal instruct) >ii L s 
ended. Consequently, factors which effect long-range retenMm are 
crucial to the design of any truly effective instruction pro^iam. Ihe 
previously mentioned hierarchical organization of contrived verl 1 
behaviois constitutes one such factor. There are two other raj* 
factors with which we should be concerned. 

First, during the early stages of learning we may frequently iind 
it necessary to introduce various degrees and forms of guidance (hints, 
prompts, and analogies, for instance) as a means of obtaining the 
production of those behaviors that we want the student to learn. Such 
guidance will n;>t be present in the job situation, and consequently 
it is necessary that the student learn to produce the behavior without 
the guidance. Therefore, such guidance must be removed or faded fr 
succeeding practice trials, and it may need to be faded at different, 
races for different students. Simply fading guidance from the instruc- 
tional presentation may in itself not be sufficient. Such guidance 
may continue to exist as unobservable behavior in the student for short 
periods of time. Thus, it is necessary not only to fade guidance from 
the instructional presentations, but also to fade it from the student's 
immediate memory. 

Guidance provides a means for controlling student response immedi- 
ately preceding and during the actual production of each response. 
Immediate Knowledge of results, or feedback, provides a method for 
controlling student response immediately after the production of each 
response. Feed^uck to the student concerning the adequacy of each 
response or sets of responses immediately after its production serves 
two purposes : 



(1) If the response was incorrect, feedback serves as a cue 
for trying e different response to the same situation 
and may provide additional guidance to facilitate occur- 
rence of the correct response. 

(2) If the response was correct, feedback serves as a cue 

to terminate further response in that situation and sets 
the stage foi the occurrence of a new situation. 



If the student does not produce the correct response or set of 
responses for a given situation, he should not be allowed to proceed 
to another situation requiring a different response. He must produce 
the correct response for each situation before proceeding in the 
instructional progrrm, even though obtaining the production of a cor- 
rect response in the first situation may require the introduction of 
veiy heavy guidance. £nch guidance, in turn, should be faded from 
ensuing similar instructional situations. Feedback should be provided 
for each practice m it rather than for each behavioral omponent within 
a practice unit. Note, however, that the practice uni' may vary in 
size, depending cn the nature of the material, the intelligence and 
previous education of the student, and the particular stage in practice 
at which the student has arrived. 
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Second, irrespective of different opinions concerning the basic 
nature of learning, I am sure that we will all agree that practice is 
one crucial factor affecting learning. One major problem in designing 
an instructional program, however, is that of integrating the various 
practice schedules for each of the different kinds of behaviors that 
we want the students to learn. For irstance, in employing distributed 
practice in an instructional program, we are faced with the problem 
of providing activities for the studert to perform during the periods 
between practice trials. The selecticn and design of these inter- 
polated activities may be more critics 1 to the learning of a given 
behavior than is the determination of a maximally efficient temporal 
practice schedule for that behavior. Although some general principles 
for solving this kind of problem do e*ist in the psychological liter- 
ature, they have not been sufficiently well spelled-out to preclude 
the need for a la^rge portion of "art." 



Minimizing I nterference With L earn ing 

There are three major sources of iiterference with learning in 
instruction programs. 

(1) Inadequate instructional ictivities. 

( 2 ) Improper administrative aid management practices. 

(3) Students' interpersonal problems. 

In every instance the interference occurs as behaviors that are 
incompatible with the occurrence of beiaviors that the student needs 
to learn. In the first instance, such incompatible behaviors consist 
of incorrect behavioral responses to virious instructional presenta- 
tions. For reasons of efficiency and effectiveness, we want to guard 
against the occurrence of incorrect behavioral responses throughout the 
entire course of learning. Some have objected to stringent application 
of this principle, claiming that there are times when it is desirable 
for studentj to learn to distinguish between correct and incorrect 
responses to a situation. This contention is quite valid, but irrele- 
vant. It should be noted that there iu a difference between learning 
an incorrect behavioral response and learning that a behavioral response 
is incorrect. The occurrence of incorrect responses can be reduced to 
the pcint of being virtually negligible through the adroit use of various 
forms of guidance in the early stages of learning and by employing prac- 
tice units of a proper size in each staje of learning. Practice units 
should generally be small in the early stages of learning and become 
progressively larger in the later stages. 

Incorrect responses in other than negligible amounts can detract 
from the effectiveness and efficiency o : an instructional program 
because of their incompatibility with the correct responses. Incorrect 
responses, however, are not the only behaviors which may occur during 
an instructional program that are incompatible with the occurrence of 
the behaviors that the student needs to learn. Other incompatible 
behaviors in the form of tension anxiet) , or worry can be introduced 
into the instructional situation by improper administrative and manage- 
ment practices. For instance, such a situation can arise if the 
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instructor terrorizes his students with a dread of coming examinations, 
or if a student enters an instructional program as a means of preparing 
for a career field and then finds that his chances ror placement in his 
chosen career field are small. Factors beyond the direct control of 
the school may also lead to the occurrence of incompatible behaviors 
during instruction. For instance, the student may have financial 
troubles, or he may have any variety of interpersonal problems which 
produce behaviors that interfere with those required for successful 
completion of the instruction program. Many of the factors leading to 
the production of incompatible behaviors may be eradicated or at least 
alleviated through proper counseling and guidance services. Such 
services should be an integral part of the management structure of 
all large instructional institutions, for they can and should contrib- 
ute to the effectiveness and efficiency of instruction. 

III. EVALUATION 

There are two major aspects to instructional evaluation. The first 
is evaluating the effectiveness of the instruction itself in inculcat- 
ing students with the behaviors selected for them to learn. The second 
is evaluating the effectiveness of the behaviors selected for the student 
to learn for producing adequate job performance. Hus, the first is con- 
cerned with the effectiveness of the instruction!;! situations and their 
arrangement for producing improvement in learning with respect to the 
purpose of instruction, while the second is concerned with the adequacy 
of the instruct ional purpose itself for producing adequate job performance. 

Basically, a test is a procedure for gathering data about specified 
behaviors in order to make a decision about those behaviors. The first 
step in constructing an achievement test for evaluating the effectiveness 
of the instruction is that of determining the kinds of decisions or 
actions that are to be based upon the outcome of the test. Thus, the 
tests or data-gathering procedures should be tailored to the decisions 
that will be made on the basis of such data. If we are constructing a 
major end-of-phase or end-of -course examination, we will be primarily 
concerned with determining those students who have learned the specified 
behaviors and those who have not, that is, stuoents who are to pass and 
students who are vo fail. The fact that some passing students learn 
more than other passing students is of little concern to us unless we 
are going to base some action on this difference. There is little point 
in differentiating between students unless they are to be differ- 
ently treated. 

Ideally successful instruction should produce students all of whom 
make a perfect score on the achievement tests; that is, the resulting 
distribution should have a variance of zero and a mean equal to the 
maximum possible score. This, o* course, is an ideal which is virtually 
impossible to realize. However, the mean and variance of the distribu- 
tion of the achievement test scores should be a function of the adequacy 
of the selection procedures and instructional activities, but not a 
requisite characteristic of the test. Instructors hiving become overly 
concerned with such things as item difficulties, interitera correlations, 
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nd the shape of the distribution of scores. Not one of these is a 
relevant test characteristic for ideally successful instruction. By 
far the most important characteristic of a test is the validity of 
the behaviors required for its successful completion to adequate per- 
formance in the ultimate job situation. From the point of view of 
rhe behaving organism (t.e,, the student who goes on to perform the 
iob), we want to maximize the degree of congruence between the behav- 
iors ho practices during instruction, the behaviors required of him 
for successful completion of the examinations, and the behaviors 
required of him for successful job performance, 

A large proportion of the behaviors practiced by students during 
instruction and required of them for successful jcb performance are 
not normally available to external observation, but used to be made 
observable for evaluation purposes. In making unobservable behaviors 
observable for evaluation purposes, it is extremely important that 
we not change significantly the behavior or the conditions in which 
it occurs. We rust be careful to make only those minimum changes 
required for miking the behavior observable. Unobservable behaviors 
nay sometimes he inferred to have occurred on the basic of the occur- 
rence of an observable behavior. For instance, in many instruct iona 1 
situations our concern primarily with the student’s unobservable 
veibal behavior (i.e . , ’‘thinking") which we might assess hy means of 
a multiple-choice test. We assume that the student's placing an X 
beside the correct choice in an item was immediately preceded by the 
occurrence of a desired verbal behavior; that is, he "thought’ r the 
problem through properly. Our concern is not with the student’s 
v-making behavior, but rather with the unobservable verbal behaviors 
rfhich we infer to have occurred by the placement of the X. The ade- 
quacy cf this type of measurement depends primarily upon our being 
able to justify this infeience. 

The construction of achievement tests should not be left to the 
arbitrary whims of individual instructors. The criterion for student 
learning should be clear]y stated in a statement of instructional 
Dbjectives. Besides clearly specifying the behaviors to be possessed 
by the students at the termination of instruction, it should tell the 
Instructor how to go about deciding at that time which students will 
pass and which v 1 i 1 1 fail, Curel on^ has made this point quite veil in 
stating that: M lhe de freto aims c l an educational program, and of 

every part thertof, consist of those acts on the basis of which the 
students and the program are in fact evaluated. If any stated aim is 
not analyzed into specific actions ar.d those actions observed and 
scored and reported, the statement is ro more than empty verbiage.” 

The statement of instructional objectives should contain a specifica- 
tion of the behaviors that will be evaluated at the end of instruction, 
end all the behaviors that are in fact evaluated at the end of instruc- 
tion should be clearly specified in the statement of instructional 
object ives . 
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A statement of instruct ional objectives can be viewed as constitut- 
ing a prescription for constructing an achievement test, or it may, in 
some instances, actually be an achievement test. Ebel, 1 for instance, 
has suggested that one way by which instruct ional objectives might be 
clearly specified is by building an extended series of test items to 
be used as a basis for evaluating achievement in a program of instruc- 
tion before the program of instruction itself is built. 

The second major aspect of evaluation, that is, evaluating the 
effectiveness of behaviors selected for the student to learn for pro- 
ducing adequate job performance, is accomplished, basically, by deter- 
mining the adequacy of job performance of students who have successfully 
completed the instructional program. For jobs that already exist, such 
evaluation may be accomplished by means of job-sample proficiency tests, 
observations of on-the-job performance, supervisory evaluations, and 
so forth. For jobs that do not already exist and which may not exist 
fer some time to come, the problem is exceedingly more difficult and, 
for th< most part, has not been resolved. 

CONCLUSION AND IMPLICATIONS 

My primary effort hds been tc present a conceptual view of instruc- 
tion a-j a system for controlling student behavior so as to modify it to 
conform to a predetermined pattern. Major emphasis has been given tc 
the problem of determining the nature of the terminal behavior pattern 
with respect to the behaving organism itself. Secondary emphasis has 
been given to the problems of determining the size and composition of 
the practice unit, the integration of practice schedules, and the min- 
imization of interference with learning. Evaluation has been treated 
as a decision-making process concerned with the adequacy of the instruc- 
tional activities for modifying student behavior and the adequacy or the 
terminal behavior pattern for producing the necessary job-performance 
capabilities, 

This paper presents nothing more than c. broad conceptual framework 
to serve as a tool fer identifying research ^nd implementation problems, 
and a general approach ro the solution of these problems. The major 
goal toward fchich research efforts need to be oriented is that of 
developing rationales and procedures for constructing and conducting 
mass instructional programs. Such goals cannot be realized by develop- 
mental projects whose primary purpose is the construction of an instruc- 
tional program as a consumer product. The development of consumer 
products and the development of technological procedures cannot be 
equally emphasized in the same project. One will invariably have to 
be sacrificed to the other, usually to the detriment of technologi- 
cal development. 



^.L. Ebel, "Obtaining and Reporting Evidence on Content Validity, M 
Eduo . Psych, l. Measurement, vol. 16, no. 3, 19S6, pp* 269-282. 



72 



68 



SOME RESEARCH NEEDS IN SELECTING AND 
TRAINING PROGRAMERS 1 

William H. Melching 



The full potentialities of automated instruction can be 
achieved only if competent programers can be selected 
and trained. Personality components such as relatively 
high intelligence, interests, attitudes, and flexibility 
are discussed. 

A wave of enthusiasm has accompanied the birth and growth of a 
special activity called "automated instruction" or "programed learning." 

e enthusiasm has been so great that there is hardly an agency or 
institution engaged in education or training that has not been intrigued 
by the potentialities of automated instruction. Modern-day visionaries 
foresee marvelous developments in this area. 

Upon the preparation of appropriate programs and the development of 
reliable automatic teaching devices, increased masses of students car. 
master unlimited amounts of study material. All individuals, irrespec- 
tive of differences in intellectual capacity and experience, can, as if 
by magic, be brought to some common level of performance. 1 do not 
intend to imply, of course, that the automatic teaching device is per- 
ceived as a panacea by all individuals interested in the area of pro- 
gramed instruction. Many are approaching the area with reserve and 
caution. Nonetheless, some enthusiasts would seem to be proclaiming 
that although men may not be created equal, they may be made equal. 

The need for a revolution in education and training has betn 
succinctly expressed by Ramo (1). "We are in rapid transition today 
to a new world which threatens to be dominated by technological 
advance. . . . This rup’.d and potentially dislocating scientific 
advance can be expected to heighten . . . the coming crisis in educa- 
tion . . . Our technical growth is paralleled by social maladjustments 
still left over from previous eras . . . Education should be at the 
head of the list for priority attention . . . Khat is needed is a 
technique of education which is in keeping with the world ahead." 

Ramo then proceeds tc describe a "modern" school of the future— the 
only kind of educational system that can hope to narrow the gap between 
human social knowledge and technical knowledge. Ramo conceives of 
a highly automated program of instruction where machines are in very 
sensitive and responsive communication with the student. Classrooms 

1 Paper for Symposium, "Some Research Needs in Programmed Learning 
and in Training Programmers at Texas Psychological Association, 

Dallas, Texas, December, |961. 
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are elaborately equipped with push-button apparatus sufficient for the 
efficient teaching of all kinds of knowledge and skills. The inter- 
active capacity of this special teaching machine is significant, for 
the machine behaves like an inexhaustible private tutor. 

It is obvious from this brief description that Ramo is suggesting 
that advanced technical knowledge should be used in constructing the 
devices to provide the educational skills that are necessary, in turn, 
to utilize the technical knowledge. 

To accomplish this feat, Rumo conceives the possibility of a new 
profession called "teaching engineer" which would be concerned with 
this educational process. One aspect would relate to the design of 
the machines, and another would focus on the design of the material 
for the machines. 

Because of the tremendous advances in technical knowledge it may 
be assumed that the design and the development of' the kinds of machines 
envisioned by Ramo are well within reasonable expectations, and that 
their production could proceed with few major problems. 

The second task of the teaching engineer-the design of material 
for the machines-is uot as easily resolved, for the procedures to be 
followed are not explicit. The mechanics of constructing the machine 
are more readily teachab.e, apparently, than are the mechanics of 
preparing the program for use on the machine. Since it seems a truism 
to say that the machine is only as good as the program to be used on 
it, and since programs must be prepared by programers, it follows that 
the full potentialities of automated instruction can be achieved only 
if programers can be trained to accomplish their tasks adequately. 

An important question is now po:;ed: What constitutes a good pro- 
gramed or to put it another way, how does one become a good programed 
The literature is not much help in these problems. There is frequent 
reference to programing as an art an 1 to recommended techniques to be 
used by programers. But there is a paucity of information directed 
at describing the good programer, or what sorts of training might be 
most profitable in producing good programers. The^r selection and 
training is apparently just as much an art as is programing itself. 

Nonetheless, there is a general recognition that programing may 
require unique skills. For example, Komoski (2) says: "Those who 

believe the schools should not do programming say that such pro- 
gramming will inevitably result in a great many poor programs, 
because they will be created by peopJe who do not have the tine, 
talent, training, or temperament requisite to the programming art," 

Let us consider some characteristics of programers that may be of 
importance, and some areas where research into programer character- 
istics might prove fruitful. 

In a relatively early report Donald Smith (3) said: "One of the 

most striking similarities among the successful programmers with whom 
I have worked is an inverted style of thinking. The inverted thinker 
focuses so intensively upon a stimulus configuration, either percep- 
tual or conceptual, that the differentiation process appears to 



continue far beyond that of the normal. Such a person tend? to be ana- 
lytical; deductive, methodical, perfect ionistic— in short, the classical 
or Jungian introvert. * 

In another instance, Lysaught (4j presented some comments centering 
around selection and training of programers. He reported that he found 
no easy way to anticipate or predict who will be a good programer. In 
selecting programmers for computer operators, it was found that the 
greatest correlation existed between intelligence and program skill. 

Yet the most intelligent individuals also became bored with programing. 
In selecting individuals to program instructional materials, Lysaught 
narrowed the concentration to those persons whose prime qualification 
was a deep and abiding interest in the work. These individuals believed 
in programing and were buoyed up by seeing results of their work. 
Lysaught noted that programing proceeded more rapidly in some areas 
than in others, and although oo precise results were given, he stated 
that the difference lay in programers rather than in subject matter. 

If these twe reports have any validity, we are left with a picture 
of the successful programer as an intelligent introvert-but not too 
intelligent. In our attempts to program at the Air Defense HujTan 
Research Unit at Fort Bliss, Texas, we have arrived at conclusions 
similar to those previously expressed. For example, one prospective 
programer portray, -d by his attitude that he did not believe in the 
programing of material to be learned, The result, of course, was 
simple; he produced little useful programing. 

Other attitudes of programers may be equally detrimental to effec- 
tive programing. If the. programer possesses the attitude that pro- 
graming is extremely d.fficult, and that it requires extended periods 
of reflection intermixed with minutely detailed analyses of each possi- 
ble step, his production is painfully slow. He is saying, in effect, 
"Programing is an extremely long and laborious task." And by his work 
output he proves that he is right. 

Tne description of the successful programer may furn out to be 
something like this: He is a relatively intelligent individual who 

accepts the basic premise that the programing of material to be learned 
is an effective means by which to teach individuals. He openly 
expresses interest in programing, he believes that he can contribute 
materially in this activity, and he sets goals for himself that are 
both realistic and realizable. He is, on the one hand, flexible enough 
to modify his program or his ideas when necessary while, on the other 
hand, he is sufficiently compulsive to be able to bring his program to 
fruition. His primary motivation for these activities is internal. 

Although this description is not as precise as or as definitive 
as one might like, it may provide some useful guidance for research. 

A cursory examination of the description suggests several personality 
components of interest: 

Relatively high intelligence 
Interests in the area 

Attitudes favorable to the area and favorable to 
achieving the goal 



( 1 ) 

( 2 ) 

(3) 
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(4) Flexibility 

(5) Compulsivity 

(6) Functional level of motivation 

This list of characteristics of the hypothetical programer is cer- 
tainly not exhaustive; there is no mention, for example, of knowledge 
of subject matter area, education and experience, effective writing 
.kills, and so forth. A more comprehensive description would include 
these factors, plus several others. 

1 would suggest, however, that certain of the factors listed will 
assume greater significance than others. Thus the ability to express 
oneself clearly and concisely, the ability to organize and to specify 
educational objectives in accordance with a prescribed theoretical 
framework, and so forth, may be skills that are lost if the programer, 
by his attitude toward his task, demonstrates an unwillingness to move 
from the theoretical level to an empirical level. Effective writing 
skill carries little weight if the programer seems unable to arrive at 
the precise objectives he seeks— and therefore never gets around to 
writing frames! 

The situation mentioned above is perhaps exaggerated, but this was 
done intentionally because it highlights the dangers involved in becom- 
ing somewhat obsessed with those activities customarily thought of as 
preliminary to frame-writing. The goal of programing is to produce 
programs, not to demonstrate why frames cannot be written. 

I ara now i.n danger of having belabored the point. I am saying that 
attitudes of programers toward their work may be of singular importance, 
and that they represent a fertile area in need of inquiry and research. 

I will now deal briefly with two or three other '’factors," The 
suggestion that the programer needs to be both flexible and compulsive 
may appeal paradoxical. But unless he is able to "give up one of my 
best frames," and still direct his efforts toward completing his task, 
he cannot hope to reach the stage where he can observe his program 
teach real students-the persons for whom the program was intended. 

We have currently adopted a procedure for indoctrination of program- 
ers at Fort Bliss that may provide the beginner with rapid feedback on 
his first programing attempts, and at the same time facilitate an early 
evaluation of his potential as a programer. The procedure is simply 
that of assigning him the task of programing a relatively restricted and 
finite bit of subject matter as his first activity. We have not had 
sufficient experience with this approach to be able to evaluate it fully, 
but the consensus of the other programers is that it will be most useful. 

If one accepts the basic premise that progress in programing is pri- 
marily a function of the programer rather than of the content of the pro- 
gram, one conclusion seems clear. Agencies and institutions contemplating 
the preparation of programs must recognize at the outset that the selec- 
tion and training of programers is equally as important as are decisions 
about what material shall be programed, what kinds of mechanical devices 
shall be vied, and what modes of presentation are most useful. 
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RESEARCH PROBLEMS RELATED TO THE IMPLEMENTATION 
OF PROGRAMED INSTRUCTION 1 



Robert G. Smith, Jr. 

This paper points out the necessity for certain kinds 
of research for data pertinent to the decision process, 
selection, training, and supervision of programers, 
and to the management of the learning process. Research 
is also needed in the areas of student motivation, 
disciplinary management of students, and instructor 
scheduling, and in the use of simulators, training 
devices, and equipment as part of the training program. 

INTRODUCTION 

Many of the initial research studies conducted in the field of 
programed instruction might be termed "demonst ration studies.” These 
studies demonstrated that significant improvements in learning could 
be brought about through the use of programei instruction. Next 
there was a series of studies concerned with attempts to identify 
variables affecting the learning process within the framework of 
programed instruction. 

Recently, however, there has been an upsurge in activity designed 
to implement programed instruction 3nd to bring it out of a purely 
research and experimental phase. The purpose* of my paper will be to 
point out that those organizations which are called upon to assist 
others in the actual use of programed instruction are encountering 
a series of problems concerning wh*cn reseaich data are greatly 
recced. One of the frustrating aspects of the present situation is 
thr.t although a researcher must obtain some of the needed information 
purely m the course of preparing a program, some of this information 
is seldom reported. 

THE DECISION PROCESS IN PROGRAMED INSTRUCTION 

Data are needed relevant to the process of making the decision 
whether to use or not to use programed instruction. Most practical 
training people are quite aware of the fact that one seldom gets some- 
thing for nothing. This is particularly true in programed instruction. 
In programing you increase the expense of the preparation of a training 
program, with the hope of obtaining a significant gain in proficiency 
or a reduction in time. 
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Unfortunately, it is difficult to estimate t ie amcunt of time that 
will be saved by the use of programed instruction The published data 
vary considerably. It is not surprising that this variation should 
exist, because we usually know very little about the quality of the 
conventional course or about thu quality of the Drogram against which 
the conventional course uas prepared. It would )e desirable if some 
rigorous work could be done to identify rather clearly those situa- 
tions in which the greatest reduction in time cojld be expected from 
programed instruction, as opposed to those situations in which little 
reduction could be expected. Data on the amount of gain in final 
achievement or the amount of time saved as affec.ed by various factors 
would be very significant information to have when one is facing a 
skeptical training executive. 

Let us suppose that we have convinced our sloptical training execu- 
tive that possibly he could use programed instru:tion. The next logical 
question is, where to get a program? We tell hi ,i that he can buy a 
program already developed by somebody and available for sale, or he can 
hire a contractor to build a program just for him, or he can develop a 
programing staff and build his own. 

With regard to purchasing programs already a^ailabl^, there is a 
need, not for research, but for the development of appropriate profes- 
sional standards. In the present burgeoning and changing state of the 
art, it might be premature to set standards that all programs have to 
meet, but it is not too carl> to consider preparing specifications of 
the information that should be made available to the prospective pur- 
chaser of a program. If we had professionally developed standards for 
reporting certain kinds of information about programs, it would make 
the decision of whether to use "off the shelf" procurement much easier. 
At a minimum, it would appear desirable to have ^ statement of the 
objectives of the program, a statement of the intended student popula- 
tion, and data concerning results that have been obtained by means of 
the program. 



It is, of course, quite likely that our training executive will 
not find a commercially available program to mee: his particular 
requirements. So he may consider the possibility of contracting for 
the preparation of a specially designed program. Again, he is faced 
with a decision process. How is he to know that a particular contrac- 
tor is likely to deliver a useful product? Admittedly, programing 
practices vary widely, but the outlines of certa n kinds of consist- 
ency are beginning to emerge, at least in the practices of the few 
groups with which I am familiar. It would seem tuite reasonable to 
require that a contractor provide his prospective client with a com- 
plete description of the processes by which he plans to develop the 
programs so that they may be jud^nd against whatever conventions the 
current state of the art suggests constitute good practice. But 
suppose our training executive says, "I don’t trest outsiders to 
build programs for me; I think I had better get ny own programing 
staff." At this point he is acquiring a large ni/nbcr of problems 
on which we have very scanty data. The requirement for information 
here is extensive. 
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THE PROBLEMS OF MANAGING A PROGRAMING STAFF 



Selection 

The earliest programs were usually prepared by people who were 
enthusiastic about the possibilities of programed instruction and who 
prepared programs covering material they already knew. These people 
usually turned out programs in what might be considered a reasonable 
amount of time. It is the comm n experience of people who have pro- 
graming staffs to discover with a shock that not everybody can program. 
In some instances this may have to be restated as "Everybody won't 
program," Eiased on extremely limited experience, it would appear that 
the major variable involved might well be attitudinal. At the present 
time, about the only selection system that is used consistently is the 
"hire, try, and fire" system. Of course nearly everybody we have 
talked to admits that this is not very efficient; some of them have 
developed t ho i r own hypotheses about how to select programers, but we 
really know very little. Shop talk among programing groups is full of 
such statements as, "A good programer is where you find him." 

Irajnj ng 

Currently most people appear to be learning how to program by means 
of a kind of apprenticeship. They become generally knowledgable in the 
field of programing by means of reading, discussions, work shops, and 
so forth, and then try their hand at preparing programs. It seems to 
take two to four months of this type of activity before the programer 
becomes reasonably proficient. A real difficulty in training stems 
from the fact that the most effective teachers of the programer are 
the students on whom he tries out his draft program. Research is needed 
to develop effective and quicker methods of obtaining programing pro- 
ficiency. Obviously, before we can improve the proficiency of pro- 
gramors by improved training methods, we need to knew how to measure 
the proficiency of programers. How do we tell a gojd programer from 
one who is not quite so good? 

I have mentioned that one hypothesis about thr selection of pro- 
gramers is that they should have appropriate attitudes. We have found 
that certain kinds of attitudes appear to interfere with the process 
of developing programs quickly. One attitude is expressed by the view 
that the individual already knows how to teach something, and doesn't 
quite see the need for all this programing rigmatole. Of course we 
do not know whether this attitude is a positive factor in poor perform- 
ance or whether it is a rationalization for poor performance, but we 
do know that we need to have effective techniques for overcoming atti- 
tudes that appear to interfere with the process of developing pro- 
graming ski 1 1 . 

Supe rvi sion 

Most programers appear to find it difficult to work steadily for 
eight hours a day. Nevertheless , unless some control is exercised 
over the nature and duration of breaks, a supervisor may find that a 
programer who has struck a slow and difficult period will be interfering 



with the progress of those programers who happen to be suddenly at a 
stage where it is quite easy for then to turn out many frames. We 
need to have reports on the accumulated experience of groups with 
programing staffs. At the present time, case histories would be 
most welcome, although it is hoped that as the field develops there 
will be experimentation. 

Another supervisory problem concerns the internal editing and 
review of the program before it is ready for testing on students. It 
seems obvious that certain kinds of editing for consistency are desir- 
able, as well as review for the accuracy of technical content. At the 
same time, it is quite clear that continued interval review quickly 
reaches a point of diminishing returns. Successive polishing of a 
program may make it communicate very well with other programers, but 
may interfere in communication with students. 

There is one source of data available from everyone who programs, 
but which not everyone reports. This is information concerning pro- 
duction rates. I do not mean production rates just for the writing 
of frames, because this is only one ? spect of the preparation of the 
program. Extremely useful information for estimating the cost of 
preparation of programs, even on a "hall park basis," could be 
obtained if people would report how long it took them to accomplish 
different aspects of the programing process. How many man-hours 
were spent ii; preparing clear object .ves, how many in preparing a 
criterion test, in outlining the consent to be presented in the pro- 
gram and determining its sequence, in writing the frames, in pretesting 
the program, in program revision? 

RESEARCH PROBLEMS IN THE MAN AG :MENT OF THE LEARNING PROCESS 

Another need for research is creited by the tremendous differences 
between programed instruction, with its emphasis on individual rates of 
progress, and conventional instruction. Most training organizations 
have had many years in which to work out the administrative and other 
management problems associated with scheduling classes of specified 
length. They have had very little experience with courses in which 
the student progresses at his own rate. We need research studies aimed 
at developing efficient techniques of both intrinsic and extrinsic 
student motivation, disciplinary maivgement of students, instructor 
scheduling, and the problems involved with potential bottlenecks when 
expensive simulators, training devices, or actual equipment are used 
as part of a training program. 

In summary, recent developments in the use of programed instruction 
have created a need for certain kinrs of research. We need data perti- 
nent to the decision process, selection, training, and supervision of 
programers, and to the management oi the learning orocess. 



PROGRAMMED INSTRUCTION— WHERE WE ARE TODAY IN THE MILITARY 1 
William H. Melching 



Acceptance and application of programmed instruction to 
the training problems of the Air Force and the Army are 
discussed in this paper. Programmer training workshops 
are described. 



In general, the military services have shown a somewhat cautious 
interest in the possible application of programmed instruction to their 
training problems. In some instances reactions have ranged from unal- 
loyed enthusiasm to open antagonism, but, by and large, the prevailing 
attitude has been positive. 

On the basis of my experience, it seems easier to interest Army 
school instructors in the use of programmed instruction than to inter- 
est many college educators in its use. That this casual observation 
portends anything significant is doubtful, of course. Perhaps the 
military is more willing to accept the fact that there is a continuing 
need to improve training, while educators are interested in education— 
not in training. 

When one searches the literature to find studies or experiments 
conducted by the military in which programmed instruction has been 
compared with another method of instruction, or in which experimental 
studies of programming variables have been involved, very few such 
e;f{ rts are found* Most of the work in this area to date has been 
accomplished by nonmilitcry agencies. 

At the same time, however, military and military research agencies 
have produced many theoretical and survey articles about programmed 
instruction. Let me list six as a br^ef sample: 

Carr, Self- Instructional Devices: A Review of Current Concepts , 

Wright-Patterson AFB, August 1959. 

Ekstrand, et al Teaching Machines in the Modem Military 
Organization , Kright-Patterson AFB, August 1960. 

Ugelow, Motivation and the Automation of Training: A Litera- 

ture Review , Wright-Patterson AFB, March 1962. 



j l Paper for Symposium "Programmed Instruction— Where Ke Are Today." 

I at Texas Psychological Association meeting in San Antonio, Texas, 

December 1962, 
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Darby, An Annotated Bibliography on the Automation of Instruc- 
tion ... HiunRRQ Research Memorandum, July 1959. 

Silverman, Automated Teaching: A Review of Theory and Research; 

Naval Training Device Center, June 1960. 

Marine Corps Equipment Board, Automatic Teaching Machine , 

' Device s; Interim Report (Part I) > Quantico, Virginia, 

July 1961. 

I think these reports are encouraging evidence of the early interest 
of the military in this new technique. I would suggest that a movement 
toward implementation is now in progress, and that we can expect the 
military to become an ever-increasing consumer of programed learn- 
ing techniques. 

Let us look at some specific activities. At this time I have no 
information about implementation efforts within the Navy, so my remark 1 
will be limited to the Air Force and the Army. 

We could probably begin at no better place than with Colonel 
Ofiesh's statement of the Air Force program in programmed instruction. 1 
Under its Air Training Command, the Air Force has undertaken a study 
of the technique in two phases. 

In Phase I, programmed learning techniques were to be applied on a 
restricted and limited basis to current training problems within the 
Air Force. Accompanying this was the desire to develop a limited in- 
house capability in programmers. On the basis of replies from a ques- 
tionnaire sent to over 140 institutions, companies, and individuals 
claiming competence to instruct in programming, contractors v;sre 
selected and formal courses lasting two to three weeks were initiated. 

Students were required to work in teams of two or three, each 
class consisting of approximately 20 students. Subject matters to be 
programmed were assigned to the students prior to entrance into the 
course, and only one of three basic approaches (linear, brandling, or 
mathetics) was taught in any one class. 

After the course, the students returned co their home bases and 
continued work on their programs. During the next few months, stu- 
dents met with thoir course instructors for additional assistance in 
the development of their programs. Once completed, each program is 
being subjected to experimental test to determine its effectiveness 
in comparison with the conventional method of instruction. 

As of September 1962, approximately 200 Air Force personnel and 
civilians had received training in programming, and some 100 programs 



^Gabriel D. Ofiesh. "The Air Training Command Program in Programmed 
Instruction," Headquarters Air Training Command, Randolph AFB, Texas. 
Paper presented to: (1) the Armed Forces Section of the Department of 

Audio-Visual Instruction, National Education Association, Kansas City, 
March 1962; (2) at a Symposium at the Southern Society for Philosophy 
and Psychology at Memphis, April 1S62. 
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were being written. Field tests of these programs will occur soon. 

The Air Force plans to train an additional 100 programmers, at which 
time the first phase will be completed. 1 

Phase II will consist of an expanded utilization of the capabili- 
ties developed in Phase I, and the extent of the expansion will be 
dependent on the success achieved in the first phase. The primary 
criterion of success, of course, will be measured in terms of the 
efficiency of the programs developed. 

In summary then, we can see that the Air Force has sought to 
explore the technique in a limited and controlled way so that final 
judgment can be deferred until sufficient data are obtained. Certainly 
by the end of Phase I, it will have developed a sizable in-house 
capability for programming. Upon attainment of positive results, the 
Air Force should be in an enviable position to obtain full benefit 
from its experiences. 

Let us turn to the Army where there has been 3 gradual increase 
in interest in programmed instruction during approximately the past 
four years. A large portion of Army effort has been exerted through 
the Human Resources Research Gffice. Several research units within 
HumRRO have initiated exploratory work in this area, but research under 
TEXTRUCT at Fort Bliss, Texas, has been particularly concerned with 
evaluating the feasibility of programmed instruction for Army techni- 
cal training. 

Recently, while this research was under way, program contractors 
and manufacturers of teaching machines began approaching Army schools 
with offers of programs, devices, anc so forth, and it became apparent 
that Army personnel would be likely to need information about pro- 
grammed instruction to enable tnem to evaluate these offers. At about 
the same time, an interest in the possibility of developing an in- 
house capability for Army programming began to arise, and this led 
quite naturally to the decision to let HumRRO offer a series of work- 
shops. Accordingly, at Fort Bliss plans were developed for a two-day 
orientation workshop for supervisory ;evel personnel, and a two-week 
workshop to train programmers; each of these has now been conducted 
four times. More than ICO students have completed the programmer 
training workshop. 

It may be interesting to note some of the differences and similari- 
ties between the programmer training workshops ot the two services. 
Course length has been approximately the same, and the number of stu- 
dents in attendance at any one class has also been approximately equal. 
Army students have been encouraged to work in teams, but this, has not 
always been possible. The subject matters brought for programming by 



l Gabriel D. Ofiesh, f, Developing an In-house Capability in Pro- 
grammed Instruction for Large Scale Systems, M Headquarters Air Training 
Command, Randolph AFS, Texas. Paper presented at a Synposiura at the 
American Psychological Association, St. Louis, September 1962. 
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Army students were assigned by their respective department or division 
heads, but it is not known whether the students were instructed to 
complete their programs following the workshop. 

Whereas Air Force students received instruction in only one pro- 
gramming approach, each Army workshop introduced three. Primary emphasis 
was placed upon linear programming, but branching was also introduced, as 
w>?ll as a variation of linear programming we Lave called looping* Each 
student was free to attempt whatever approach most appealed to him. The 
particular approach was usually based on the student's evaluation of his 
subject matter, the anticipated variation in background of his students, 
and his own preferences. 

It was not unusual for students to exhibit considerable ingenuity in 
this situation, reflecting in part their reluctance to adhere to only 
one approach. Also, some students employed one technique during the 
workshop, but indicated that they intended to use another when they 
returned to their home posts. Students were encouraged to initiate 
requests for consultations when they felt the need for assistance with 
their work following formal classes. 

The decision to train additional programmers will rest directly 
within the Army, and HumRRO will probably not continue its workshops. 

As a sequel, however, the Fort Bliss tfnit will attempt to automate por- 
tions of its workshop for training programmers, and this program will 
become available to Army schools. 

In addition to the HumRRO work in programmed instruction already 
described, several independent programming endeavors have been under- 
taken within the Army school system. The same general situation may 
well prevail in the other service branches. 

At the U.S. Army Signal School at Fort Monmouth, New Jersey, pro- 
grams in both Direct and Alternating Current have been prepared and are 
being used on a trial b3sis. A program in Communication Procedures has 
been developed at Fort Gordon, Georgia, and the programming of additional 
course material is under way at this time. The Ordnance School at 
Aberdeen, Maryland has developed branching programs in Supply and Stock 
Levels, and current plans call for a continuation of t hi r effort. 



Each Department of the Air Defense School at Fort Bliss has been 
directed to prepare programs covering two hours of selected regular 
course work. The Basic Electronics Department of this school has com- 
pleted a program in Direct Current and is presently constructing a 
program in Alternating Current. 



It is significant that most military reports on the topic of pro- 
grammed instruction do rot deal with the question: "Is programmed 

instruction an efficient and effective method of instruction?" but 
rather ask: "How can programmed instruction be effectively imple- 

mented?" In other words, there seems to be rather widespread accept- 
ance of the technique as an effective instructional tool, The difficult 
tasks ceni er around the selection and training of programmers, the 
development of programs, and the integration of program packages into 
ongoing training systems. 
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In summary I would suggest that programmed instruction in the 
military today has passed the initial exploratory stage and is now 
on the threshold of implementation* The speed with which implemen- 
tation will occur is debatable, but interestingly enough, seems to 
be not too dependent or a demonstration of great success in initial 
studies. This conclusion is easily reached by reading summaries of 
some earlier studies* Even though the results may not be highly 
supportive of programming — that is, even when programmed instruction 
is only equal to or partly superior to conventional instruct ion— the 
reports invariably and with an expressed intent to continue work in 
the area. Military researchers are certain that programmed instruc- 
tion must not be prematurely discarded. 
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APPENDIX 1 



THE AUTOMATION 
OF 

INSTRUCTION 

R.R . Ridenour 

Task TEXTRUCT 

U S. ARMY AIR DEFENSE HUMAN RESEARCH UNIT 
FORT BLISS, TEXAS 



The purpose of this pamphlet is to acquaint you with a new 
method of instruction, quite different from any you previously 
havebeen exposed to. In the box below there are five questions 
important to your understanding of learning and automated 
instruction. After you have read the remaining pages in this 
pamphlet, you will be able to answer these questions andunder- 
stand how important they are to the field of education. 



I. What is automated instruction! 

II, What is the purpose of automation studies'? 

III. What are three of the most important principles 
of learning? 

IV. How can these principles be applied? 

V. Ho. might automated instruction affect the student and 
the teacher? 



*Tbis brochure by Richard R. Ridenour was distrib- 
uted at the International Science Exhibit, El Paso, 
Texas, March 1960. A device for teaching meter-reading 
was displayed and "troop-tested" by HumRRO. 
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I. WHAT IS AUTOMATED INSTRUCTION? 



The term "automated instruction" or "teaching machine" 
is used to describe a method or device that can act as a pri- 
vate tutor for an individual student without the need for 
continual attention from a human instructor. In this way, 
each learner is provided v/ith just th<? information he needs at 
the time he needs it, thereby allowing him to progress at his 
own best rate of speed, Automated instruction is a procedure 
which insists that the s tudent , rather than the teacher, is the 
most important element in the learning situation and it is 
designed to change the learner from a passive receiver of 
informationto an active participant in the process of learning. 



II. WHAT IS THE PURPOSE OF AUTOMATION STUDIES? 

The reason for the recent concern with instructional 
automation methods is that the science of learning is quite 
well established hut is not, at present, effectively used to the 
full extent of its potential. It has become increasingly appar- 
ent that it is impossible to apply the science of learning most 
effectively ir, the classical classroom situation. In the class- 
room, instruction must be geared for a group of different 
individuals instead of being aimed at the single student and 
varied according to his own needs. Thus, research on auto- 
mation is an attempt on the part of the researchers to learn 
how to apply the actual science of learning in the most effec- 
tive manner possible. 



III. WHAT ARE THREE OF THE MOST IMPORTANT 

PRINCIPLES OF LEARNING? 

1) The Principle of Participation 

This principle states that the learner must actively 
engage, or participate, in the learning. We all know that a 
person "learns by doing." What does this actually mean? 
Reading, listening, or watching are doing "doing” but in a 
passive rather than active way. The learner must participate 
actively by being forced to think and act by answering questions 
after each bit of information presented. Ke must practice those 
activities which ho is expected to learn. It has been experi 
mentally shown that active participation normally leadsto more 
effective learning. 

2) The Principle of Immediate Knowledge of Results 

This, our second principle, is really two principles in 
one: the principle of knowledge of results and the principle of 
immediacy. By this we mean that the learner must know 
whether or not his answer to a question is correct and he must 
also know immediate ly. A lapse of even a few seconds following 
the answering of a question, and before the result of the answer 
is known, often leads to ineffective learning. Thus, it is essen- 
tial to design a learning environment in which knowledge of 
results may be provided immediately. 

3) The Principle of Individual Differences 

In Ihe same way that people differ with respect to 
height, weight, etc., they also differ as to the rate at which 
they learn . Since some people learn faster and some slower, 
material must be sequenced and presented according to the 
needs of the individual learner, if effective instruction is to 
be achieved. 

There are, 0 : course, many other established principles 
of learning. How^.vei , the above three are a sample of some of 
the most import int. 
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IV. HOW CAN THESE PRINCIPLES LE APPLIED? 



These principles cannot be applied effectively a the 
typical classroom. It is necessary to design a new kird of 
learning environment in which the principles can be applied. 

There arc several ways to apply the principles of 
effective learning. One way is the teaching machine. Another 
way is the specially sequenced or programmed textbook. The 
main problem facing researchers is not which “machine” (or 
instructional method) to use. Rathei , it is how to present, or 
sequence, the subject matter in the most effective way. 



V. HOW MIGHT AUTOMATED INSTRUCTION AFFECT 

THE STUDENT AND THE TEACHER? 

1) The Student 

Introduction of automated teaching in the school 
curriculum would tend to give the learner a feeling of person- 
alized instruction, tailored to his own needs. By this we mean, 
a student would be allowed to learn at his own rate and at his 
own convenience. Materiel piesented would depend upon 
answers to previous questions. For these reasons, it is to be 
assumed that automated instruction would tend to greatly 
increase student motivation and desire to learn by taking away 
some of the typical drudgery of study and replacing it by inter- 
esting individualized Instruction. 

2 ) The Teacher 

Teaching devices will not in any way replace the human 
teacher. What it will do is to free the teacher from much 
routine instruction, thereby allowing him to give individual 
help to slower students and to assist faster students in their 
pursuit of more extensive Knowledge of the subject. Due to this 
change inhls job, teacher prestige shoulo be greatly enhanced. 



I 

The following is a short quU to see how well YOU have 
understood the principles of learning and effective teaching. 
Try answering the following questions in order to see if our 
explanation has been clear. You will find the ar swers on the 
next two pages. Check vour answer on pages 6 and 7 after 
each question and before going on to the next. 

1. Which is more effective for learning, active •‘thinking" participation 
or passive listening, watching, or reading? 

a) Active 

b) Passive 

(Turn to pages 6 and 7 for the correct answer.) 

2. Which of the following is most likely to assure immediate knowledge 
of right or wrong? 

a) A lecture 

b) A small class 

c) A privet.* tutor 

3. If a test or examination is to be effective as a le arniig experience , 
when is the best time to give the learner knowledge of results? 

a) As soon ai he answers each question 

b) As soon as he has finished the test 

c) As soon as the teacher has finished grading and recording 
the papers 

4. To insure understanding and prevent boredom on the part of the 
learner we must direct our teaching at the indivldue k Which of the 
following does this best? 

a) A lecture 

b) A teaching device 

c) A small class 

5. From the standpoint of effective learning, which of the following is 
the best way to present material? 

a) Television 

b) A lecture 

c) Individual "active" Instruction 

d) A textbook 

S. Which of the following is the most effective way to ‘each? 

a) A large lecture 

b) A small class 

c) A "good" prlvat tutor, machine or otherwise j 
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ANSWERS 



1. You certainly answered “active participation” is the more 
effective because the learner is forced to 'think" and « 
understand the material presented. 
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1, Machine or “device" teaching requires the learner to 
actively participate by asking questions immediately 
following each bit of information. 



2. When automated, the "mi.chine” asks questions and the 
student must answer before proceeding. Some machines 
have been devised where ;he student can ask questions of 
his own. But the principle of "active participation" can 
be fulfilled by the machir.3 doing the asking. 



3. Teaching devices are designed so that the learner is 
always informedof the accuracy of his responses as soon 
as he makes them. 



4. A “machine" teaching program can present material as 
it is required by the individual student. The question of 
what information should < ome next is determined by how 
well the student answers the questions. 



5. Automated teaching give ? individual active instruction 
and places the emphasis on the individual and the science 
of learning. 



6. The ''tutor" combines the principlesof learning with an 
effective learning environment to emphasize-learning. 
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SUMMARY AND CONCLUSIONS 



The idea of automating instruction is not new (original 
patent taken out 1566) but real interest in the great teaching 
possibilities inherent Ln the concept has only been evidenced in 
the past three years. In such a short period of time the 
research has only served to pi mote interest and demonstrate 
the vast possibilities of combining subject material and the 
science of learning. Research is aimed not at deciding what 
subjects to teach but at how to teach them. In this pamphlet we 
have been mainly concerned with the scientific principles of 
learning and the possibility of their application to au r omated 
instruction. It is important to remember that although the 
machine contributes to an effective learning environment, the 
most important aspect of automated instruction is not the 
"gadgetry" but the learning material v.bich is placed within the 
machine and the way it is presented. The motl spectacular 
machine is of absolutely no value unless the "program" of 
instruction has been carefully and accurately constructed to 
best fit the needs of the individual learner. All "machines" 
and the material to put in them are in a strictly experimental 
stage. However, it is expected that great changes will be made 
in the next few years which will radically advance the educa- 
tional situation 'in our country. The thought is not to replace 
our institutions of learning or our teachers; the idesT is^to 
increase their teaching capabilities through the application of 
scientific learning methods. We are living in a continually 
changing world, yet education is still much the same as it was 
a hundred years ago. Automated teaching offers a great possi- 
bility of bringing education up to date. 
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OTHER PUBLICATIONS UNDER WORK UNIT TEXTRUCT 



An Annotated Bibliography on the Automation of Instruction Research 

Memorandum by Charles D. Darby, July 1959. PB-159959 AD-228 766 

An Evaluation of an Experimental Meter Reading Trainer, Research 
Memorandum by Robert G. Smith, Jr. and Richard R. Ridenour, 

October 1960. AD-815861L 

Results of Exploratory Investigations Conducted for the Purpose of 
Planning a Research Pribram on Instructional Methods, Research 
Memorandum, March 1961- AD-253 395 

Teaching Machines and Programmed Instruction — Some Factors to Consider 
in Implementation, Research Memorandum by Robert G. Smith, Jr., 
August 1961. AD-632 188 

A Procedural Guide to the Programming of Instruction: Prelimirtary 

Report, Research Memorandum by William H. Melching, March 1962. 

AD-279 569 

The Text of an Orientation Workshop in Automated Instruction, Consulting 
Report by William H. Melching, John A. Cox, Jesse C. Rupe, and 
Robert G. Smith, Jr., July 1962. AD-637 117 

Studies of Fixed Procedures Training: A Preliminary Test of Self- 

Instructional Method, Research Memorandum by Paul G. Whitmore, 

July 1963. At 1 - 420 453 

A Handbook for Programmers of Automated Instruction, procedural guide 
by William H. Melching, Robert G. Smith, Jr., Jesse C. Rupe, and 
John A. Cox, September 1963. AD-632 558 

Evaluation of an Auto- Instructional Program on the First Week of a 

Basic Electronics Course, Research Memorandum (revised) by William 
H. Melching, Harold E, Christensen, and Albert L. Kubala, 

March 1964. AD-60] 681 

The following research materials were among those prepared under Work 

Unit TEXTPUCT: 

Pocketechool Series, manuals for experimental teaching program (pub- 
lished by U.S Army Air Defense School), July I960.' 

Mathematics I, Multiplication and Division (Decirrale) : 

Part One, Part Two, Part Three 

Mathematics II 3 Multiplication and Division (Cancellation) 



Volume IX and Supplemental Graph Book were published by Division 
No. 5 (Air Defense), June 1963. 
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Mathematics III , Powers and Roots: 

Part One, Part Two 

Mathematics IV , Powers of Ten 

Mathematics V, Simp l a Equations; 

Part One, Part Two, Part Three 

Mathematics VI > Proportions: 

Part One, Part Two 

Mathematics VII , Stated Problems: 

Reference Items for Parts One and Two 
Part One, Part Two, Part Three 

Mathematics VIII 3 Nomograms 

Mathematics IX> with 

Supplemental Graph Book 
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